Rust

The official Rust SDK for Spidra lets you extract structured data from any website by describing what you want in plain English. It handles JavaScript rendering, anti-bot bypass, and CAPTCHA solving as a managed API, so your Rust code stays focused on the data. Built on tokio and reqwest, the SDK uses native Rust async/await throughout and returns Result<T, SpidraError> on every call.

Installation

Add the dependency to your Cargo.toml:

[dependencies]
spidra = "0.1"

Or via Cargo:

cargo add spidra

Get your API key from app.spidra.io under Settings → API Keys. Keep your key out of source control — read it from an environment variable.

Getting started

All requests require an API key sent as the x-api-key header. Pass it to the client:

let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").expect("SPIDRA_API_KEY not set"));

Quick start

use spidra::{SpidraClient, types::ScrapeParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").unwrap());

    let result = client
        .scrape()
        .run(&ScrapeParams::new("https://example.com"))
        .await?;

    println!("{}", result.content.unwrap_or_default());
    Ok(())
}

run() submits the job and polls until it completes, then returns the result.

Scraping

Scrape a single page

use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeParams::new("https://news.ycombinator.com");
    params.output_format = Some(OutputFormat::Markdown);
    params.render_js = Some(true);

    // run() polls until complete; submit() returns immediately with a job ID.
    let result = client.scrape().run(&params).await?;

    if let Some(content) = result.content {
        println!("{content}");
    }
    Ok(())
}

AI extraction with a prompt

use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeParams::new("https://news.ycombinator.com");
    params.output_format = Some(OutputFormat::Json);
    params.prompt = Some("Extract the top 5 story titles and their URLs".to_string());

    let result = client.scrape().run(&params).await?;

    if let Some(data) = result.data {
        println!("{}", serde_json::to_string_pretty(&data).unwrap());
    }
    Ok(())
}

Submit and poll manually

If you need more control over polling (e.g. to show progress), submit and poll separately:

use spidra::{SpidraClient, types::ScrapeParams};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    // submit() returns immediately with a job ID
    let queued = client
        .scrape()
        .submit(&ScrapeParams::new("https://example.com"))
        .await?;

    println!("Job submitted: {}", queued.job_id);

    // Poll manually
    loop {
        let status = client.scrape().get(&queued.job_id).await?;
        println!("Status: {}", status.status);

        match status.status.as_str() {
            "completed" => {
                if let Some(result) = status.result {
                    println!("{}", result.content.unwrap_or_default());
                }
                break;
            }
            "failed" | "cancelled" => {
                eprintln!("Job ended with status: {}", status.status);
                break;
            }
            _ => tokio::time::sleep(Duration::from_secs(3)).await,
        }
    }
    Ok(())
}

Job statuses: waiting · active · completed · failed · cancelled

Custom polling options

Use run_with_options to override the default poll interval and timeout:

use spidra::{SpidraClient, types::ScrapeParams};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");
    let params = ScrapeParams::new("https://example.com");

    let result = client
        .scrape()
        .run_with_options(&params, Duration::from_secs(2), Duration::from_secs(60))
        .await?;

    println!("{}", result.content.unwrap_or_default());
    Ok(())
}

Batch scraping

Submit up to 50 URLs in a single request. All URLs are processed in parallel.

use spidra::{SpidraClient, types::{BatchScrapeParams, ScrapeUrl, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let urls = vec![
        ScrapeUrl::new("https://example.com"),
        ScrapeUrl::new("https://example.org"),
        ScrapeUrl::new("https://example.net"),
    ];

    let params = BatchScrapeParams::new(urls);

    // run() submits and blocks until the whole batch finishes.
    let response = client.batch().run(&params).await?;

    println!(
        "Completed: {} / Failed: {}",
        response.completed_count, response.failed_count
    );

    for item in &response.items {
        if let Some(result) = &item.result {
            println!("{}: {} chars", item.url, result.content.as_deref().unwrap_or("").len());
        }
    }
    Ok(())
}

Item statuses: pending · running · completed · failed Batch statuses: pending · running · completed · failed · cancelled

Crawling

Give Spidra a starting URL and instructions for which links to follow. It discovers pages automatically and extracts structured data from each one.

use spidra::{SpidraClient, types::CrawlParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = CrawlParams::new("https://docs.example.com");
    params.max_pages = Some(50);
    params.max_depth = Some(3);
    params.include_paths = Some(vec!["docs/**".to_string()]);

    let pages = client.crawl().run(&params).await?;

    println!("Crawled {} pages", pages.len());
    for page in &pages {
        println!("  {} — {}", page.status, page.url);
    }
    Ok(())
}

Logs

Every API scrape job is logged automatically.

use spidra::{SpidraClient, types::ScrapeLogsParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeLogsParams::default();
    params.limit = Some(20);
    params.status = Some("completed".to_string());

    let logs = client.logs().list(&params).await?;
    for log in &logs.logs {
        println!("{} — {} — {:?} credits", log.uuid, log.url, log.credits_used);
    }
    Ok(())
}

Usage statistics

Returns credit and request usage broken down by day or week.

use spidra::{SpidraClient, types::UsageRange};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let rows = client.usage().get(UsageRange::ThirtyDays).await?;
    for row in &rows {
        println!("{}: {} requests, {:.2} credits", row.date, row.requests, row.credits);
    }
    Ok(())
}

Range	Description
`UsageRange::SevenDays`	Last 7 days, one row per day
`UsageRange::ThirtyDays`	Last 30 days, one row per day
`UsageRange::Weekly`	Last 7 weeks, one row per week

Error handling

All methods return Result<T, SpidraError>. The error variants are:

Variant	When
`SpidraError::Auth`	401 / 403 — invalid or missing API key
`SpidraError::RateLimit`	429 — rate limit exceeded
`SpidraError::InsufficientCredits`	402 — not enough credits
`SpidraError::Api { status, message }`	Other 4xx errors
`SpidraError::ServerError { status, message }`	5xx server errors
`SpidraError::Timeout { seconds }`	Polling timed out
`SpidraError::JobFailed { message }`	Remote job reached failed state
`SpidraError::JobCancelled`	Remote job was cancelled
`SpidraError::Http(e)`	Underlying `reqwest` transport error
`SpidraError::Json(e)`	JSON deserialisation error

match client.scrape().run(&params).await {
    Ok(result) => println!("{}", result.content.unwrap_or_default()),
    Err(spidra::SpidraError::Auth) => eprintln!("Invalid or missing API key"),
    Err(spidra::SpidraError::RateLimit) => eprintln!("Rate limited — back off and retry"),
    Err(spidra::SpidraError::InsufficientCredits) => eprintln!("Out of credits"),
    Err(spidra::SpidraError::Timeout { seconds }) => {
        eprintln!("Timed out after {}s", seconds)
    }
    Err(e) => eprintln!("Error: {e}"),
}

Java

Official Java SDK — CompletableFuture-based, builder pattern, no extra HTTP dependencies.

Overview

Official SDKs

Installation

Getting started

Quick start

Scraping

Scrape a single page

AI extraction with a prompt

Submit and poll manually

Custom polling options

Batch scraping

Crawling

Logs

Usage statistics

Error handling

Java

Overview

Official SDKs

​Installation

​Getting started

​Quick start

​Scraping

​Scrape a single page

​AI extraction with a prompt

​Submit and poll manually

​Custom polling options

​Batch scraping

​Crawling

​Logs

​Usage statistics

​Error handling

Java

Installation

Getting started

Quick start

Scraping

Scrape a single page

AI extraction with a prompt

Submit and poll manually

Custom polling options

Batch scraping

Crawling

Logs

Usage statistics

Error handling