> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Rust

> Official Rust SDK for Spidra — AI-powered web scraping with proxy rotation and CAPTCHA handling.

The official Rust SDK for Spidra lets you extract structured data from any website by describing what you want in plain English. It handles JavaScript rendering, anti-bot bypass, and CAPTCHA solving as a managed API, so your Rust code stays focused on the data.

Built on `tokio` and `reqwest`, the SDK uses native Rust async/await throughout and returns `Result<T, SpidraError>` on every call.

## Installation

Add the dependency to your `Cargo.toml`:

```toml theme={null}
[dependencies]
spidra = "0.1"
```

Or via Cargo:

```sh theme={null}
cargo add spidra
```

<Note>
  Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**.
  Keep your key out of source control — read it from an environment variable.
</Note>

***

## Getting started

All requests require an API key sent as the `x-api-key` header. Pass it to the client:

```rust theme={null}
let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").expect("SPIDRA_API_KEY not set"));
```

## Quick start

```rust theme={null}
use spidra::{SpidraClient, types::ScrapeParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").unwrap());

    let result = client
        .scrape()
        .run(&ScrapeParams::new("https://example.com"))
        .await?;

    println!("{}", result.content.unwrap_or_default());
    Ok(())
}
```

`run()` submits the job and polls until it completes, then returns the result.

***

## Scraping

### Scrape a single page

```rust theme={null}
use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeParams::new("https://news.ycombinator.com");
    params.output_format = Some(OutputFormat::Markdown);
    params.render_js = Some(true);

    // run() polls until complete; submit() returns immediately with a job ID.
    let result = client.scrape().run(&params).await?;

    if let Some(content) = result.content {
        println!("{content}");
    }
    Ok(())
}
```

### AI extraction with a prompt

```rust theme={null}
use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeParams::new("https://news.ycombinator.com");
    params.output_format = Some(OutputFormat::Json);
    params.prompt = Some("Extract the top 5 story titles and their URLs".to_string());

    let result = client.scrape().run(&params).await?;

    if let Some(data) = result.data {
        println!("{}", serde_json::to_string_pretty(&data).unwrap());
    }
    Ok(())
}
```

### Submit and poll manually

If you need more control over polling (e.g. to show progress), submit and poll separately:

```rust theme={null}
use spidra::{SpidraClient, types::ScrapeParams};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    // submit() returns immediately with a job ID
    let queued = client
        .scrape()
        .submit(&ScrapeParams::new("https://example.com"))
        .await?;

    println!("Job submitted: {}", queued.job_id);

    // Poll manually
    loop {
        let status = client.scrape().get(&queued.job_id).await?;
        println!("Status: {}", status.status);

        match status.status.as_str() {
            "completed" => {
                if let Some(result) = status.result {
                    println!("{}", result.content.unwrap_or_default());
                }
                break;
            }
            "failed" | "cancelled" => {
                eprintln!("Job ended with status: {}", status.status);
                break;
            }
            _ => tokio::time::sleep(Duration::from_secs(3)).await,
        }
    }
    Ok(())
}
```

**Job statuses:** `waiting` · `active` · `completed` · `failed` · `cancelled`

### Custom polling options

Use `run_with_options` to override the default poll interval and timeout:

```rust theme={null}
use spidra::{SpidraClient, types::ScrapeParams};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");
    let params = ScrapeParams::new("https://example.com");

    let result = client
        .scrape()
        .run_with_options(&params, Duration::from_secs(2), Duration::from_secs(60))
        .await?;

    println!("{}", result.content.unwrap_or_default());
    Ok(())
}
```

***

## Batch scraping

Submit up to 50 URLs in a single request. All URLs are processed in parallel.

```rust theme={null}
use spidra::{SpidraClient, types::{BatchScrapeParams, ScrapeUrl, OutputFormat}};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let urls = vec![
        ScrapeUrl::new("https://example.com"),
        ScrapeUrl::new("https://example.org"),
        ScrapeUrl::new("https://example.net"),
    ];

    let params = BatchScrapeParams::new(urls);

    // run() submits and blocks until the whole batch finishes.
    let response = client.batch().run(&params).await?;

    println!(
        "Completed: {} / Failed: {}",
        response.completed_count, response.failed_count
    );

    for item in &response.items {
        if let Some(result) = &item.result {
            println!("{}: {} chars", item.url, result.content.as_deref().unwrap_or("").len());
        }
    }
    Ok(())
}
```

**Item statuses:** `pending` · `running` · `completed` · `failed`

**Batch statuses:** `pending` · `running` · `completed` · `failed` · `cancelled`

***

## Crawling

Give Spidra a starting URL and instructions for which links to follow. It discovers pages automatically and extracts structured data from each one.

```rust theme={null}
use spidra::{SpidraClient, types::CrawlParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = CrawlParams::new("https://docs.example.com");
    params.max_pages = Some(50);
    params.max_depth = Some(3);
    params.include_paths = Some(vec!["docs/**".to_string()]);

    let pages = client.crawl().run(&params).await?;

    println!("Crawled {} pages", pages.len());
    for page in &pages {
        println!("  {} — {}", page.status, page.url);
    }
    Ok(())
}
```

***

## Logs

Every API scrape job is logged automatically.

```rust theme={null}
use spidra::{SpidraClient, types::ScrapeLogsParams};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let mut params = ScrapeLogsParams::default();
    params.limit = Some(20);
    params.status = Some("completed".to_string());

    let logs = client.logs().list(&params).await?;
    for log in &logs.logs {
        println!("{} — {} — {:?} credits", log.uuid, log.url, log.credits_used);
    }
    Ok(())
}
```

***

## Usage statistics

Returns credit and request usage broken down by day or week.

```rust theme={null}
use spidra::{SpidraClient, types::UsageRange};

#[tokio::main]
async fn main() -> Result<(), spidra::SpidraError> {
    let client = SpidraClient::new("your-api-key");

    let rows = client.usage().get(UsageRange::ThirtyDays).await?;
    for row in &rows {
        println!("{}: {} requests, {:.2} credits", row.date, row.requests, row.credits);
    }
    Ok(())
}
```

| Range                    | Description                    |
| ------------------------ | ------------------------------ |
| `UsageRange::SevenDays`  | Last 7 days, one row per day   |
| `UsageRange::ThirtyDays` | Last 30 days, one row per day  |
| `UsageRange::Weekly`     | Last 7 weeks, one row per week |

***

## Error handling

All methods return `Result<T, SpidraError>`. The error variants are:

| Variant                                        | When                                   |
| ---------------------------------------------- | -------------------------------------- |
| `SpidraError::Auth`                            | 401 / 403 — invalid or missing API key |
| `SpidraError::RateLimit`                       | 429 — rate limit exceeded              |
| `SpidraError::InsufficientCredits`             | 402 — not enough credits               |
| `SpidraError::Api { status, message }`         | Other 4xx errors                       |
| `SpidraError::ServerError { status, message }` | 5xx server errors                      |
| `SpidraError::Timeout { seconds }`             | Polling timed out                      |
| `SpidraError::JobFailed { message }`           | Remote job reached failed state        |
| `SpidraError::JobCancelled`                    | Remote job was cancelled               |
| `SpidraError::Http(e)`                         | Underlying `reqwest` transport error   |
| `SpidraError::Json(e)`                         | JSON deserialisation error             |

```rust theme={null}
match client.scrape().run(&params).await {
    Ok(result) => println!("{}", result.content.unwrap_or_default()),
    Err(spidra::SpidraError::Auth) => eprintln!("Invalid or missing API key"),
    Err(spidra::SpidraError::RateLimit) => eprintln!("Rate limited — back off and retry"),
    Err(spidra::SpidraError::InsufficientCredits) => eprintln!("Out of credits"),
    Err(spidra::SpidraError::Timeout { seconds }) => {
        eprintln!("Timed out after {}s", seconds)
    }
    Err(e) => eprintln!("Error: {e}"),
}
```

<CardGroup cols={1}>
  <Card title="Java" icon="java" href="/sdks/java">
    Official Java SDK — CompletableFuture-based, builder pattern, no extra HTTP dependencies.
  </Card>
</CardGroup>