# Spidra Documentation

> Technical documentation for the Spidra AI-powered web scraping API. Covers the REST API, all official SDKs, browser automation, CAPTCHA solving, proxy scraping, and workflow integrations.

## Primary URLs

- Docs home: https://docs.spidra.io
- API reference: https://docs.spidra.io/api-reference/introduction
- Quickstart: https://docs.spidra.io/quickstart
- SDK overview: https://docs.spidra.io/sdks/overview
- Main website: https://spidra.io
- App: https://app.spidra.io

## Getting Started

- Quickstart (`/quickstart`): Make your first scrape in under 2 minutes via the Playground or in code. Sign up, get an API key from Settings → API Keys, then submit a scrape job. 300 free credits on signup, no card needed.
- API Keys (`/features/api-keys`): Create and manage API keys. Use them as `Authorization: Bearer YOUR_API_KEY`.

## API Basics

- Base URL: `https://api.spidra.io/api`
- Authentication: `Authorization: Bearer YOUR_API_KEY`
- All scrape and crawl jobs are **asynchronous**: submit a job, get a `jobId` back, poll until `status` is `completed`, retrieve results from `result.content`

### Endpoints

- `POST /scrape` submits a scrape job (1-3 URLs; multi-URL content is merged into one AI extraction)
- `GET /scrape/{jobId}` polls job status
- `POST /batch/scrape` batch scrapes up to 50 URLs, each processed independently
- `GET /batch/scrape/{batchId}` polls batch status; `DELETE` cancels; `POST .../retry` re-queues failed items
- `POST /crawl` submits a crawl job
- `GET /crawl/{jobId}` polls crawl status; `GET /crawl/{jobId}/pages` returns per-page results with signed HTML/markdown URLs; `POST /crawl/{jobId}/extract` re-extracts without re-crawling; `DELETE /crawl/{jobId}` cancels
- `GET /scrape-logs` lists past jobs
- `GET /usage-stats?range=7d|30d|weekly` returns usage stats

## Core Features

### Scrape (`/features/playground`)
Submit a URL and a plain-English prompt. Spidra opens a real headless browser, executes JavaScript, runs AI extraction, and returns structured data. Output formats: json, markdown, text, table. Screenshots available via the `screenshot` flag.

### Batch Scrape (`/features/batch-scraping`)
Submit up to 50 URLs in one `POST /batch/scrape` request. All run in parallel and each URL gets its own independent result. Retry only failed items without re-running successful ones. Per-item results in the response.

### Crawl (`/features/crawling`)
Give a base URL and a crawl instruction. Spidra maps the site, follows relevant links, and extracts structured data from every page. Set `maxPages`, filter by URL patterns, and get a full dataset in one job.

### Structured Output (`/features/structured-output`)
Add a `schema` (JSON Schema) to any scrape request. The AI returns JSON that exactly matches the defined shape: correct field names, correct types, `null` for missing values, never a missing key. The Node and Python SDKs also accept Zod and Pydantic models directly as schemas. Define every field you want extracted; untyped objects with no properties come back empty.

### Browser Actions (`/features/actions`)
Add an `actions` array to a URL to interact with the page before extraction: `click`, `type`, `scroll`, `wait`, `navigate`, `waitForElement`, and `forEach`. `forEach` loops over every matching element on a page, optionally navigating to each linked page and running per-item AI extraction.

### CAPTCHA Solving (`/features/captcha-solver`)
Automatic CAPTCHA and anti-bot challenge handling. Runs transparently on every job with no API parameter needed. Supports reCAPTCHA v2/v3, hCaptcha, Cloudflare Turnstile, DataDome, and more. Billed at 10 credits per solve.

### Stealth Mode (`/features/stealth-mode`)
Routes requests through residential proxies across 50+ countries. Enable with `"useProxy": true`. Target a specific country with `"proxyCountry": "us"` (ISO 2-letter code) or use `"eu"` to rotate across EU member states. Billed against bandwidth quota, not credits.

### Authenticated Scraping (`/features/authenticated-scraping`)
Scrape login-protected pages by passing session cookies in the `cookies` field. Supports raw Cookie header format and standard browser DevTools copy-paste. Cookies are never stored server-side.

### Presets (`/features/presets`)
Save a URL, prompt, schema, and all settings as a named preset. Rerun with one click, put on a schedule, or connect to an integration.

### Integrations (`/features/integrations`)
Deliver extracted data automatically to Slack, Discord, email, Telegram, webhooks, and more. Combine with presets and schedules for fully automated recurring pipelines.

## SDKs

All SDKs are open source at https://github.com/spidra-io.

- Node.js (`/sdks/node`): `npm install spidra`. Retries with backoff, typed errors, streaming watchers, Zod schema input, webhook signature verification. Zero dependencies, runs in Node 18+, browsers, and edge runtimes.
- Python (`/sdks/python`): `pip install spidra`. Async-first with sync wrappers, retries, typed errors, watch generators, Pydantic schema input, webhook verification.
- Go (`/sdks/go`): `go get github.com/spidra-io/spidra-go`. Fully typed, zero deps.
- PHP (`/sdks/php`): `composer require spidra/spidra-php`. PHP 8.1+, Guzzle 7.
- Ruby (`/sdks/ruby`): `gem install spidra`. Zero external dependencies.
- Java (`/sdks/java`): `io.spidra:spidra-java-sdk:0.1.0`
- .NET (`/sdks/dotnet`): `dotnet add package Spidra`
- Swift (`/sdks/swift`): Swift Package Manager
- Rust (`/sdks/rust`): `cargo add spidra`
- Elixir (`/sdks/elixir`): Mix dependency

## AI and Workflow Integrations

- MCP Server (`/sdks/mcp`): `npx -y spidra-mcp` (npm package `spidra-mcp`). Connects Spidra to Claude Code, Claude Desktop, Cursor, Windsurf, and VS Code as native tools. 12 tools: spidra_scrape (1-3 URLs, one combined answer), spidra_batch_scrape (2-50 URLs, independent results), spidra_crawl (discover pages from one URL), plus status, pages, re-extract, cancel, logs, and usage tools. Configure with the `SPIDRA_API_KEY` environment variable.
- LangChain (`/sdks/langchain`): `pip install langchain-spidra` (PyPI package `langchain-spidra`). `SpidraLoader` document loader (scrape, crawl, and batch modes) for RAG pipelines, plus `SpidraScrape`, `SpidraCrawl`, and `SpidraBatchScrape` agent tools. Python only; JavaScript agents should use the Node SDK with the Vercel AI SDK.
- n8n (`/sdks/n8n`): Community node `n8n-nodes-spidra`. Trigger scrapes, batches, and crawls as workflow steps.

## Billing

Credit-based model:
- Base: 2 credits per URL scraped
- AI token usage: additional usage-based cost (input and output tokens)
- CAPTCHA solving: 10 credits per solve
- Stealth mode (proxies): billed by bandwidth quota, no per-URL credit charge

Plans:
- Free ($0/month): 300 credits
- Starter ($19/month): 5,000 credits
- Builder ($79/month): 25,000 credits
- Pro ($249/month): 125,000 credits
- Enterprise: custom

## Rate Limits

Submission limits per user account: 60 scrape jobs/minute, 20 batch jobs/minute. Polling endpoints are exempt. Exceeding the limit returns HTTP 429.