Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.spidra.io/llms.txt

Use this file to discover all available pages before exploring further.

The official Rust SDK for Spidra lets you extract structured data from any website by describing what you want in plain English. It handles JavaScript rendering, anti-bot bypass, and CAPTCHA solving as a managed API, so your Rust code stays focused on the data. Built on tokio and reqwest, the SDK uses native Rust async/await throughout and returns Result<T, SpidraError> on every call.

​
Installation

Add the dependency to your Cargo.toml: 
[dependencies]
spidra = "0.1"
Or via Cargo: 
cargo add spidra
Get your API key from app.spidra.io under Settings → API Keys. Keep your key out of source control — read it from an environment variable.

​
Getting started

All requests require an API key sent as the x-api-key header. Pass it to the client: 
let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").expect("SPIDRA_API_KEY not set"));

​
Quick start

use spidra::{SpidraClient, types::ScrapeParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").unwrap());

    let result = client
        .scrape()
        .run(&ScrapeParams::new("https://example.com"))
        .await?;

    println!("{}", result.content.unwrap_or_default());
    Ok(())
}
run() submits the job and polls until it completes, then returns the result.

​
Scraping

​
Scrape a single page

use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeParams::new("https://news.ycombinator.com");
    params.output_format = Some(OutputFormat::Markdown);
    params.render_js = Some(true);

    // run() polls until complete; submit() returns immediately with a job ID.
    let result = client.scrape().run(&params).await?;

    if let Some(content) = result.content {
        println!("{content}");
    }
    Ok(())
}

​
AI extraction with a prompt

use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeParams::new("https://news.ycombinator.com");
    params.output_format = Some(OutputFormat::Json);
    params.prompt = Some("Extract the top 5 story titles and their URLs".to_string());

    let result = client.scrape().run(&params).await?;

    if let Some(data) = result.data {
        println!("{}", serde_json::to_string_pretty(&data).unwrap());
    }
    Ok(())
}

​
Submit and poll manually

If you need more control over polling (e.g. to show progress), submit and poll separately: 
use spidra::{SpidraClient, types::ScrapeParams};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    // submit() returns immediately with a job ID
    let queued = client
        .scrape()
        .submit(&ScrapeParams::new("https://example.com"))
        .await?;

    println!("Job submitted: {}", queued.job_id);

    // Poll manually
    loop {
        let status = client.scrape().get(&queued.job_id).await?;
        println!("Status: {}", status.status);

        match status.status.as_str() {
            "completed" => {
                if let Some(result) = status.result {
                    println!("{}", result.content.unwrap_or_default());
                }
                break;
            }
            "failed" | "cancelled" => {
                eprintln!("Job ended with status: {}", status.status);
                break;
            }
            _ => tokio::time::sleep(Duration::from_secs(3)).await,
        }
    }
    Ok(())
}
Job statuses: waiting · active · completed · failed · cancelled

​
Custom polling options

Use run_with_options to override the default poll interval and timeout: 
use spidra::{SpidraClient, types::ScrapeParams};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");
    let params = ScrapeParams::new("https://example.com");

    let result = client
        .scrape()
        .run_with_options(&params, Duration::from_secs(2), Duration::from_secs(60))
        .await?;

    println!("{}", result.content.unwrap_or_default());
    Ok(())
}

​
Batch scraping

Submit up to 50 URLs in a single request. All URLs are processed in parallel. 
use spidra::{SpidraClient, types::{BatchScrapeParams, ScrapeUrl, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let urls = vec![
        ScrapeUrl::new("https://example.com"),
        ScrapeUrl::new("https://example.org"),
        ScrapeUrl::new("https://example.net"),
    ];

    let params = BatchScrapeParams::new(urls);

    // run() submits and blocks until the whole batch finishes.
    let response = client.batch().run(&params).await?;

    println!(
        "Completed: {} / Failed: {}",
        response.completed_count, response.failed_count
    );

    for item in &response.items {
        if let Some(result) = &item.result {
            println!("{}: {} chars", item.url, result.content.as_deref().unwrap_or("").len());
        }
    }
    Ok(())
}
Item statuses: pending · running · completed · failed Batch statuses: pending · running · completed · failed · cancelled

​
Crawling

Give Spidra a starting URL and instructions for which links to follow. It discovers pages automatically and extracts structured data from each one. 
use spidra::{SpidraClient, types::CrawlParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = CrawlParams::new("https://docs.example.com");
    params.max_pages = Some(50);
    params.max_depth = Some(3);
    params.include_paths = Some(vec!["docs/**".to_string()]);

    let pages = client.crawl().run(&params).await?;

    println!("Crawled {} pages", pages.len());
    for page in &pages {
        println!("  {} — {}", page.status, page.url);
    }
    Ok(())
}

​
Logs

Every API scrape job is logged automatically. 
use spidra::{SpidraClient, types::ScrapeLogsParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeLogsParams::default();
    params.limit = Some(20);
    params.status = Some("completed".to_string());

    let logs = client.logs().list(&params).await?;
    for log in &logs.logs {
        println!("{} — {} — {:?} credits", log.uuid, log.url, log.credits_used);
    }
    Ok(())
}

​
Usage statistics

Returns credit and request usage broken down by day or week. 
use spidra::{SpidraClient, types::UsageRange};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let rows = client.usage().get(UsageRange::ThirtyDays).await?;
    for row in &rows {
        println!("{}: {} requests, {:.2} credits", row.date, row.requests, row.credits);
    }
    Ok(())
}
RangeDescription
UsageRange::SevenDaysLast 7 days, one row per day
UsageRange::ThirtyDaysLast 30 days, one row per day
UsageRange::WeeklyLast 7 weeks, one row per week

​
Error handling

All methods return Result<T, SpidraError>. The error variants are:
VariantWhen
SpidraError::Auth401 / 403 — invalid or missing API key
SpidraError::RateLimit429 — rate limit exceeded
SpidraError::InsufficientCredits402 — not enough credits
SpidraError::Api { status, message }Other 4xx errors
SpidraError::ServerError { status, message }5xx server errors
SpidraError::Timeout { seconds }Polling timed out
SpidraError::JobFailed { message }Remote job reached failed state
SpidraError::JobCancelledRemote job was cancelled
SpidraError::Http(e)Underlying reqwest transport error
SpidraError::Json(e)JSON deserialisation error
match client.scrape().run(&params).await {
    Ok(result) => println!("{}", result.content.unwrap_or_default()),
    Err(spidra::SpidraError::Auth) => eprintln!("Invalid or missing API key"),
    Err(spidra::SpidraError::RateLimit) => eprintln!("Rate limited — back off and retry"),
    Err(spidra::SpidraError::InsufficientCredits) => eprintln!("Out of credits"),
    Err(spidra::SpidraError::Timeout { seconds }) => {
        eprintln!("Timed out after {}s", seconds)
    }
    Err(e) => eprintln!("Error: {e}"),
}

Java

Official Java SDK — CompletableFuture-based, builder pattern, no extra HTTP dependencies.