# Get Usage Statistics Source: https://docs.spidra.io/api-reference/account/usage-stats GET /usage-stats Retrieve time-series usage data for your account, including requests, tokens, crawls, and credit consumption over a chosen period. This endpoint returns aggregated usage metrics for your account over a selected time window. Use it to build usage dashboards, monitor credit burn rate, or understand how your workloads are distributed across days and weeks. ## Time Ranges Pass the `range` query parameter to select the window you want: | Value | Description | | -------- | ------------------------------------- | | `7d` | Last 7 days, one data point per day | | `30d` | Last 30 days, one data point per day | | `weekly` | Last 7 weeks, one data point per week | ```bash cURL theme={null} # Last 7 days curl "https://api.spidra.io/api/usage-stats?range=7d" \ -H "Authorization: Bearer YOUR_API_KEY" # Last 30 days curl "https://api.spidra.io/api/usage-stats?range=30d" \ -H "Authorization: Bearer YOUR_API_KEY" # Weekly summary (last 7 weeks) curl "https://api.spidra.io/api/usage-stats?range=weekly" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests # Last 7 days response = requests.get( "https://api.spidra.io/api/usage-stats", headers={"Authorization": "Bearer YOUR_API_KEY"}, params={"range": "7d"} ) # Last 30 days response = requests.get( "https://api.spidra.io/api/usage-stats", headers={"Authorization": "Bearer YOUR_API_KEY"}, params={"range": "30d"} ) # Weekly summary (last 7 weeks) response = requests.get( "https://api.spidra.io/api/usage-stats", headers={"Authorization": "Bearer YOUR_API_KEY"}, params={"range": "weekly"} ) ``` ```javascript Node.js theme={null} // Last 7 days const response = await fetch( "https://api.spidra.io/api/usage-stats?range=7d", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); // Last 30 days const response = await fetch( "https://api.spidra.io/api/usage-stats?range=30d", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); // Weekly summary (last 7 weeks) const response = await fetch( "https://api.spidra.io/api/usage-stats?range=weekly", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); ``` ## Response Fields Each item in the `data` array represents one time period. | Field | Type | Description | | ---------- | ------ | -------------------------------------------------------------------------------------------------------- | | `label` | string | Display label for this period, suitable for chart axes. For example: `"Mon"`, `"Dec 12"`, or `"Week 50"` | | `date` | string | The date or date range this data point covers | | `requests` | number | Number of scrape requests made in this period | | `crawls` | number | Number of crawl jobs started in this period | | `tokens` | number | Total AI tokens consumed (input and output combined) | | `captchas` | number | Number of CAPTCHAs solved in this period | | `latency` | number | Average response latency in milliseconds | | `credits` | number | Credits consumed in this period | ## Example Response ```json theme={null} { "status": "success", "data": [ { "label": "Dec 11", "date": "2025-12-11", "requests": 42, "crawls": 3, "tokens": 58400, "captchas": 6, "latency": 4100, "credits": 89 }, { "label": "Dec 12", "date": "2025-12-12", "requests": 15, "crawls": 2, "tokens": 21700, "captchas": 1, "latency": 3900, "credits": 32 } ] } ``` ## Errors If you omit the `range` parameter or pass an unsupported value, the API returns a `400` error: ```json theme={null} { "status": "error", "message": "Invalid or missing range" } ``` For a full explanation of what each metric means and how to read your usage data, see the [Usage and Analytics](/billing/usage-analytics) guide. # Submit a Crawl Job Source: https://docs.spidra.io/api-reference/crawling/crawl POST /crawl Start a crawl job that discovers and processes multiple pages from a website. Control which pages to visit, how deep to go, and what to extract from each one. ## How It Works Crawl jobs run asynchronously. You get a `jobId` immediately and poll `GET /crawl/{jobId}` until the job finishes. 1. **Submit** — Send your request, receive a `jobId` 2. **Discover** — Spidra loads your base URL and finds links matching your `crawlInstruction` 3. **Crawl** — Visits each discovered page (up to `maxPages`) 4. **Solve** — Handles any CAPTCHAs automatically 5. **Extract** — Runs your `transformInstruction` or `schema` on each page. If neither is provided, returns raw page markdown with no AI step. 6. **Poll** — Check `GET /crawl/{jobId}` until `status` is `completed` *** ## Request Fields ### Required | Field | Type | Description | | ------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `baseUrl` | string | The starting URL. Spidra begins here and follows links outward. | | `crawlInstruction` | string | Which pages to follow, in plain language. e.g. `"Crawl all blog post pages"` or `"Visit product pages only"`. Spidra uses this to decide which links to visit and which to ignore. | ### Extraction (both optional) | Field | Type | Description | | ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `transformInstruction` | string | What to extract from each page, in plain language. Applied to every page in the crawl. When omitted and no `schema` is set, each page's `data` field contains the raw page markdown — no AI is called and no token credits are charged. | | `schema` | object | A JSON Schema object defining the exact output structure. When provided, the AI returns JSON matching this schema for every page, which is useful when you need consistent, queryable results across all pages. Takes precedence over `transformInstruction` for output shape. | ### Scope and depth | Field | Type | Default | Description | | ------------------- | --------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `maxPages` | integer | `5` | Maximum number of pages to crawl (1–50). | | `maxDepth` | integer | unlimited | Maximum link depth from the base URL. `0` visits only the base URL. `1` visits the base URL and pages directly linked from it. Omit for unlimited depth. | | `includePaths` | string\[] | — | URL path patterns to include. Only pages whose paths match at least one pattern are crawled. Accepts glob-style patterns, e.g. `["/blog/*", "/news/*"]`. | | `excludePaths` | string\[] | — | URL path patterns to skip. Pages matching any pattern are not visited. e.g. `["/tag/*", "/author/*", "/login"]`. | | `allowSubdomains` | boolean | `false` | When `true`, follows links to subdomains of the base domain (e.g. `docs.example.com` when base is `example.com`). | | `crawlEntireDomain` | boolean | `false` | When `true`, follows any link on the same root domain regardless of path. Combine with `includePaths` or `excludePaths` to keep the scope manageable. | | `ignoreQueryParams` | boolean | `false` | When `true`, URLs that differ only by query string are treated as the same page. Prevents duplicate processing on sites that append tracking or session parameters to URLs. | ### Delivery | Field | Type | Description | | ------------ | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `webhookUrl` | string | A URL that receives POST requests as the job progresses. Spidra fires one event per successful page, plus a final event when the job completes. See [Webhook events](#webhook-events) below. | ### Access and authentication | Field | Type | Default | Description | | -------------- | ------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `useProxy` | boolean | `false` | Route requests through residential proxies to reduce bot detection and access geo-restricted content. | | `proxyCountry` | string | `"global"` | Two-letter ISO country code (`"us"`, `"de"`), `"eu"` for rotation across EU member states, or `"global"` for no preference. Requires `useProxy: true`. | | `cookies` | string | — | Session cookies for crawling pages that require a login. Accepts standard format (`name=value; name2=value2`) or a raw Chrome DevTools paste. | *** ## Proxy and Geo-Targeting ```json theme={null} { "baseUrl": "https://example.com/products", "crawlInstruction": "Crawl all product listing pages", "transformInstruction": "Extract product name, price, and availability", "useProxy": true, "proxyCountry": "us" } ``` Use `"proxyCountry": "global"` (or omit it) for no country preference. Use `"eu"` to rotate across EU member states. For a specific country pass its two-letter ISO code. Full country list, EU rotation, examples, and credit costs *** ## Scoped Crawling with Path Filters Combine `includePaths` and `excludePaths` to keep crawls focused on the content you actually need. ```json theme={null} { "baseUrl": "https://example.com", "crawlInstruction": "Crawl all documentation pages", "transformInstruction": "Extract the page title and main content", "maxPages": 30, "includePaths": ["/docs/*"], "excludePaths": ["/docs/changelog/*", "/docs/legacy/*"] } ``` *** ## Structured Output with Schema Use `schema` when you need every page to return the same fields in the same format — useful for feeding results directly into a database or downstream pipeline. Use the [Spidra JSON Schema Generator](https://spidra.io/tools/json-schema-generator) to build and preview your schema visually before pasting it here. ```json theme={null} { "baseUrl": "https://example.com/jobs", "crawlInstruction": "Crawl all job listing pages", "schema": { "type": "object", "properties": { "title": { "type": "string" }, "location": { "type": "string" }, "salary": { "type": "string" }, "remote": { "type": "boolean" } } }, "maxPages": 20 } ``` *** ## Raw Content (No Extraction) Omit both `transformInstruction` and `schema` to get the raw page content without any AI processing. Each page's `data` field contains the plain markdown of that page. No token credits are charged. ```json theme={null} { "baseUrl": "https://docs.example.com", "crawlInstruction": "Crawl all documentation pages", "maxPages": 20 } ``` This is useful when you want to feed the content into your own AI pipeline or process it downstream. *** ## Webhook Events When `webhookUrl` is set, Spidra sends POST requests to that URL as the job runs. All requests have `Content-Type: application/json`. **`crawl.page`** — Fired for each page that is successfully processed. ```json theme={null} { "event": "crawl.page", "jobId": "abc-123", "page": { "url": "https://example.com/blog/post-1", "title": "My Blog Post", "data": { "title": "My Blog Post", "author": "Jane Smith" } } } ``` **`crawl.completed`** — Fired once when the entire job finishes. ```json theme={null} { "event": "crawl.completed", "jobId": "abc-123", "pagesScraped": 8, "creditsUsed": 22 } ``` **`crawl.failed`** — Fired if the job fails entirely (not for individual page failures). ```json theme={null} { "event": "crawl.failed", "jobId": "abc-123", "error": "No pages successfully scraped" } ``` Webhook delivery is fire-and-forget. Spidra does not retry on failure and does not block the crawl if your endpoint is slow or unreachable. If you need guaranteed delivery, poll [GET /crawl//pages](/api-reference/crawling/crawl-pages) when the job completes. *** ## Authenticated Crawling ```json theme={null} { "baseUrl": "https://app.example.com/dashboard", "crawlInstruction": "Find all report pages", "transformInstruction": "Extract report titles and dates", "maxPages": 10, "cookies": "session_id=abc123; auth_token=xyz789" } ``` Full guide on getting cookies and formats # Cancel Crawl Job Source: https://docs.spidra.io/api-reference/crawling/crawl-cancel DELETE /crawl/{jobId} Stop a crawl job that is queued or actively running. Pages already processed are kept. Stops a crawl job that is queued or currently running. Pages that were already processed before the cancellation are preserved — you can still retrieve them with [GET /crawl//pages](/api-reference/crawling/crawl-pages). The job status moves to `cancelled`. Cancelling a job that has already reached a terminal state (`completed`, `failed`, or `cancelled`) returns a 400. ## Example Request ```bash cURL theme={null} curl -X DELETE https://api.spidra.io/api/crawl/abc-123 \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests requests.delete( "https://api.spidra.io/api/crawl/abc-123", headers={"Authorization": "Bearer YOUR_API_KEY"} ) ``` ```javascript Node.js theme={null} await fetch("https://api.spidra.io/api/crawl/abc-123", { method: "DELETE", headers: { Authorization: "Bearer YOUR_API_KEY" } }); ``` ## Response ```json theme={null} { "status": "cancelled", "jobId": "abc-123" } ``` ## Retrieving Partial Results After cancelling, call [GET /crawl//pages](/api-reference/crawling/crawl-pages) to retrieve any pages that completed before the job was stopped. ```bash theme={null} curl https://api.spidra.io/api/crawl/abc-123/pages \ -H "Authorization: Bearer YOUR_API_KEY" ``` Only pages with `status: "success"` will have data. Any pages that were mid-flight when the job was cancelled will not appear. # Get Crawl Job Details Source: https://docs.spidra.io/api-reference/crawling/crawl-details GET /crawl/job/{jobId} Retrieve the full configuration and statistics for a crawl job, including the instructions used, token consumption, and credit cost. Returns the complete record for a crawl job: the configuration you submitted, current status, token usage, and credit cost. It does not return the extracted page data. For the actual content from each page, use [GET /crawl//pages](/api-reference/crawling/crawl-pages). ## When to Use This Endpoint * Inspect the instructions and options that were used for a job * Check token and credit costs for accounting or reporting * Confirm job configuration before re-running extraction with [POST /crawl//extract](/api-reference/crawling/crawl-extract) ## Example Request ```bash cURL theme={null} curl https://api.spidra.io/api/crawl/job/abc-123 \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests response = requests.get( "https://api.spidra.io/api/crawl/job/abc-123", headers={"Authorization": "Bearer YOUR_API_KEY"} ) ``` ```javascript Node.js theme={null} const response = await fetch("https://api.spidra.io/api/crawl/job/abc-123", { headers: { Authorization: "Bearer YOUR_API_KEY" } }); ``` ## Response Fields ### Job configuration | Field | Type | Description | | ----------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `id` | string | Unique job identifier. | | `base_url` | string | The starting URL that was crawled. | | `crawl_instruction` | string | The instruction used to decide which pages to follow. | | `transform_instruction` | string or null | The prompt used to extract data from each page. `null` if extraction used a schema or if no extraction was configured. | | `output_schema` | object or null | The JSON Schema used for structured extraction. `null` if a transform instruction was used instead, or if no extraction was configured. | | `max_pages` | integer | The page limit set for this job. | | `max_depth` | integer or null | The link depth limit. `null` if not set (unlimited). | | `include_paths` | string\[] or null | Path patterns that were included. `null` if not set. | | `exclude_paths` | string\[] or null | Path patterns that were excluded. `null` if not set. | | `allow_subdomains` | boolean | Whether subdomain links were followed. | | `crawl_entire_domain` | boolean | Whether all paths on the root domain were eligible. | | `ignore_query_params` | boolean | Whether URLs differing only by query string were deduplicated. | | `webhook_url` | string or null | The webhook URL that received per-page events, if configured. | ### Status and stats | Field | Type | Description | | --------------- | -------------- | -------------------------------------------------------------------------------------------- | | `status` | string | `waiting`, `active`, `completed`, `failed`, or `cancelled`. | | `pages_crawled` | integer | Number of pages successfully processed. | | `input_tokens` | number or null | Total input tokens consumed by AI across all pages. `null` when no AI extraction was used. | | `output_tokens` | number or null | Total output tokens generated by AI across all pages. `null` when no AI extraction was used. | | `credits_used` | number or null | Total credits charged for this job. | | `created_at` | string | ISO 8601 timestamp when the job was created. | | `updated_at` | string | ISO 8601 timestamp of the last status change. | ## Example Response ```json theme={null} { "id": "abc-123", "base_url": "https://example.com/blog", "crawl_instruction": "Crawl all blog post pages", "transform_instruction": "Extract title, author, publish date, and the first 200 words of the body", "output_schema": null, "max_pages": 10, "max_depth": 2, "include_paths": ["/blog/*"], "exclude_paths": ["/blog/tag/*"], "allow_subdomains": false, "crawl_entire_domain": false, "ignore_query_params": true, "webhook_url": null, "status": "completed", "pages_crawled": 8, "input_tokens": 14820, "output_tokens": 3210, "credits_used": 25, "created_at": "2025-12-17T15:00:00Z", "updated_at": "2025-12-17T15:03:42Z" } ``` For the actual extracted content from each page, call [GET /crawl//pages](/api-reference/crawling/crawl-pages). # Download Crawl Results Source: https://docs.spidra.io/api-reference/crawling/crawl-download GET /crawl/{jobId}/download Download the results of a completed crawl job as a ZIP archive containing HTML, markdown, and extracted data files. Downloads a ZIP archive of all successfully crawled pages from a completed job. Each page is saved as one or more files inside the archive, organized by hostname and path. Use the `include` parameter to control which content types are bundled in the ZIP. ## Content Types | Value | What is included | | ---------- | ------------------------------------------------------------------------------------------------------------- | | `html` | Raw HTML file for each page | | `markdown` | Markdown version of each page | | `data` | AI-extracted data in JSON, CSV, or Markdown format (format is auto-detected from your `transformInstruction`) | If you omit the `include` parameter, all three types are included by default. ## Example Requests ```bash cURL theme={null} # Download everything (HTML, markdown, and extracted data) curl -OJ "https://api.spidra.io/api/crawl/abc-123/download" \ -H "Authorization: Bearer YOUR_API_KEY" # Download only the extracted data curl -OJ "https://api.spidra.io/api/crawl/abc-123/download?include=data" \ -H "Authorization: Bearer YOUR_API_KEY" # Download markdown and data only curl -OJ "https://api.spidra.io/api/crawl/abc-123/download?include=markdown,data" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests # Download everything response = requests.get( "https://api.spidra.io/api/crawl/abc-123/download", headers={"Authorization": "Bearer YOUR_API_KEY"} ) with open("crawl-abc-123.zip", "wb") as f: f.write(response.content) # Download only extracted data response = requests.get( "https://api.spidra.io/api/crawl/abc-123/download", headers={"Authorization": "Bearer YOUR_API_KEY"}, params={"include": "data"} ) with open("crawl-abc-123.zip", "wb") as f: f.write(response.content) ``` ```javascript Node.js theme={null} import { writeFileSync } from "fs"; // Download everything const response = await fetch( "https://api.spidra.io/api/crawl/abc-123/download", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); const buffer = await response.arrayBuffer(); writeFileSync("crawl-abc-123.zip", Buffer.from(buffer)); // Download only extracted data const dataOnly = await fetch( "https://api.spidra.io/api/crawl/abc-123/download?include=data", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); const buf = await dataOnly.arrayBuffer(); writeFileSync("crawl-abc-123.zip", Buffer.from(buf)); ``` ## ZIP Archive Structure When a single content type is requested, files are placed at the root of the archive with appropriate extensions: ``` crawl-abc-123.zip example.com_blog_post-one.json example.com_blog_post-two.json ``` When multiple content types are requested, each page gets its own folder: ``` crawl-abc-123.zip example.com_blog_post-one/ data.json index.html markdown.md example.com_blog_post-two/ data.json index.html markdown.md ``` ## Response The response is a binary ZIP file with the following headers: | Header | Value | | --------------------- | ---------------------------------------- | | `Content-Type` | `application/zip` | | `Content-Disposition` | `attachment; filename=crawl-{jobId}.zip` | Only pages with `status: "success"` are included in the download. If no successful pages exist, the API returns a 404 error. # Extract from Crawl Source: https://docs.spidra.io/api-reference/crawling/crawl-extract POST /crawl/{jobId}/extract Run a new extraction on pages from a completed crawl job without re-crawling the site This does not re-crawl the website. Spidra reads the HTML and markdown already saved from the original crawl. ## Prerequisites The source crawl job **must have a `completed` status** before you call this endpoint. Calling `/extract` on a job that is still running, pending, or failed will return a `400 Bad Request`. Poll `GET /crawl/{jobId}` and wait for `"status": "completed"` before proceeding. ## Request Body | Field | Type | Required | Description | | ---------------------- | ------ | -------- | --------------------------------------------------------------------------------------------- | | `transformInstruction` | string | Yes | The extraction prompt to apply to every page from the source crawl. Maximum 5,000 characters. | ## How It Works 1. Pass the `jobId` of a **completed** crawl job. If you ran the crawl previously, this is the `id` field shown in your crawl history. 2. Provide a `transformInstruction` describing what you want to extract. 3. Spidra loads the saved content for each page and runs your prompt against it. 4. A new crawl job is created with the results, which you can poll and download the same way as any other job. ## When to Use This * You want to extract different fields from pages you already crawled * Your first extraction prompt wasn't quite right and you want to try again * You need the same pages in two different formats, like JSON and CSV ## Polling Results The response returns a new `jobId`. Use the standard crawl endpoints to check progress and get results: | Endpoint | Purpose | | ------------------------------------ | --------------------------- | | `GET /crawl/{jobId}` | Poll job status | | `GET /crawl/{jobId}/pages` | Get extracted data per page | | `GET /crawl/{jobId}/download` | Download results as ZIP | | `POST /crawl/{jobId}/retry/{pageId}` | Retry a specific page | ## Common Errors | Status | Error message | Cause | | ------ | ------------------------------------------------------- | ------------------------------------------------------------------------------------- | | `400` | `Source crawl job has not completed successfully` | You called `/extract` before the source job finished. Wait for `status: "completed"`. | | `422` | `Missing required field: transformInstruction` | The request body is missing the `transformInstruction` field. | | `422` | `transformInstruction must be 5000 characters or fewer` | Your prompt exceeds the 5,000 character limit. | | `403` | `You have exceeded your monthly credit limit.` | Not enough credits remaining. Check your usage at `GET /usage`. | | `404` | `Source crawl job not found` | The `jobId` does not exist or does not belong to your account. | # List Crawl History Source: https://docs.spidra.io/api-reference/crawling/crawl-history GET /crawl/history Retrieve a paginated list of your past crawl jobs, including status, page counts, and credit usage. Use this endpoint to browse all the crawl jobs your account has submitted. Each record shows the base URL, how many pages were crawled, the current job status, and how many credits were consumed. This is useful for building dashboards, auditing your usage, or picking up a job ID you want to work with in a follow-up request. ## Pagination The response is paginated. Use the `page` and `limit` query parameters to navigate through your history. ```bash cURL theme={null} # First page (default: 10 results) curl https://api.spidra.io/api/crawl/history \ -H "Authorization: Bearer YOUR_API_KEY" # Second page with 25 results per page curl "https://api.spidra.io/api/crawl/history?page=2&limit=25" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests # First page response = requests.get( "https://api.spidra.io/api/crawl/history", headers={"Authorization": "Bearer YOUR_API_KEY"} ) # Second page with 25 results per page response = requests.get( "https://api.spidra.io/api/crawl/history", headers={"Authorization": "Bearer YOUR_API_KEY"}, params={"page": 2, "limit": 25} ) ``` ```javascript Node.js theme={null} // First page const response = await fetch("https://api.spidra.io/api/crawl/history", { headers: { Authorization: "Bearer YOUR_API_KEY" } }); // Second page with 25 results per page const response = await fetch( "https://api.spidra.io/api/crawl/history?page=2&limit=25", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); ``` ## Response Fields | Field | Type | Description | | ---------------------- | ------- | ---------------------------------------------------------------------- | | `jobs` | array | List of crawl job records for this page | | `jobs[].id` | string | Unique job ID. Use this to call other endpoints like GET /crawl//pages | | `jobs[].base_url` | string | The starting URL that was crawled | | `jobs[].status` | string | Job status: `waiting`, `active`, `completed`, or `failed` | | `jobs[].max_pages` | integer | The maximum number of pages requested | | `jobs[].pages_crawled` | integer | Actual number of pages successfully crawled | | `jobs[].created_at` | string | ISO 8601 timestamp when the job was created | | `jobs[].credits_used` | number | Credits charged to your account for this job | | `total` | integer | Total number of crawl jobs in your account | | `page` | integer | The current page number | | `totalPages` | integer | Total number of pages available | ## Example Response ```json theme={null} { "jobs": [ { "id": "abc-123", "base_url": "https://example.com/blog", "status": "completed", "max_pages": 10, "pages_crawled": 8, "created_at": "2025-12-17T15:00:00Z", "credits_used": 25 }, { "id": "def-456", "base_url": "https://store.example.com/products", "status": "failed", "max_pages": 20, "pages_crawled": 3, "created_at": "2025-12-16T09:30:00Z", "credits_used": 8 } ], "total": 34, "page": 1, "totalPages": 4 } ``` To get the full extracted data for a completed job, call [GET /crawl//pages](/api-reference/crawling/crawl-pages) using the `id` from this response. # Get Crawled Pages Source: https://docs.spidra.io/api-reference/crawling/crawl-pages GET /crawl/{jobId}/pages Retrieve all pages from a completed crawl job, including extracted data and signed URLs to the raw HTML and markdown files. Returns every page processed by a crawl job. Call this once the job status is `completed`. Each page record includes the extracted content in `data`, plus signed URLs to the original HTML snapshot and markdown version stored by Spidra. ## Example Request ```bash cURL theme={null} curl https://api.spidra.io/api/crawl/abc-123/pages \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests response = requests.get( "https://api.spidra.io/api/crawl/abc-123/pages", headers={"Authorization": "Bearer YOUR_API_KEY"} ) ``` ```javascript Node.js theme={null} const response = await fetch("https://api.spidra.io/api/crawl/abc-123/pages", { headers: { Authorization: "Bearer YOUR_API_KEY" } }); ``` ## Response Fields | Field | Type | Description | | ----------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pages` | array | All pages processed by this job, including failed ones. | | `pages[].id` | string | Unique page ID. Pass this to [POST /crawl//extract](/api-reference/crawling/crawl-extract) to re-run extraction on a specific page. | | `pages[].url` | string | The URL of this page. | | `pages[].title` | string | Page title as detected during crawling. | | `pages[].status` | string | `success` or `failed`. | | `pages[].data` | any | Extracted content for this page. When a `transformInstruction` or `schema` was provided, this contains AI-extracted structured data. When neither was set, this is the raw page markdown — no AI was used. | | `pages[].error_message` | string or null | Error details when `status` is `failed`. | | `pages[].html` | string or null | Signed URL to the raw HTML snapshot. Valid for 1 hour. | | `pages[].markdown` | string or null | Signed URL to the markdown version of this page. Valid for 1 hour. | | `pages[].created_at` | string | ISO 8601 timestamp when this page was processed. | ## Example Response ```json theme={null} { "pages": [ { "id": "page-uuid-1", "url": "https://example.com/blog/how-to-scrape", "title": "How to Scrape the Web Without Getting Blocked", "status": "success", "data": { "title": "How to Scrape the Web Without Getting Blocked", "author": "Jane Smith", "published": "2025-11-20", "summary": "A guide to rotating proxies and handling JavaScript-heavy pages." }, "error_message": null, "html": "https://storage.spidra.io/signed/...", "markdown": "https://storage.spidra.io/signed/...", "created_at": "2025-12-17T15:02:10Z" }, { "id": "page-uuid-2", "url": "https://example.com/blog/javascript-rendering", "title": "JavaScript Rendering Explained", "status": "failed", "data": null, "error_message": "AI transformation failed: content too short to extract", "html": null, "markdown": null, "created_at": "2025-12-17T15:02:55Z" } ] } ``` ## The `data` Field What `data` contains depends on how you configured the job: * **With `transformInstruction`** — `data` is whatever the AI extracted based on your prompt. It could be a string, an object, or structured JSON depending on what you asked for. * **With `schema`** — `data` is a JSON object matching the schema you defined, with all fields present. * **With neither** — `data` is the raw page markdown. No AI was involved and no token credits were charged. The `html` and `markdown` URLs point to the same content in its original format. ## Handling Failed Pages Pages with `status: "failed"` still appear in the response. The `error_message` explains what went wrong. To re-run extraction on a failed page without re-crawling the site, use the [extract endpoint](/api-reference/crawling/crawl-extract). The `html` and `markdown` URLs expire after one hour. If you need permanent access to the raw files, use [Download Crawl Results](/api-reference/crawling/crawl-download) to get a full ZIP archive. # Get Crawl Job Status Source: https://docs.spidra.io/api-reference/crawling/crawl-status GET /crawl/{jobId} Poll a running crawl job for progress. Returns the full result set when the job completes. Poll this endpoint after submitting a crawl job. While the job is running it returns progress information. Once the job completes, the `result` array contains one entry per successfully crawled page. Poll every 2–5 seconds. Stop when `status` is `completed`, `failed`, or `cancelled`. ## Example Request ```bash cURL theme={null} curl https://api.spidra.io/api/crawl/abc-123 \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests, time while True: r = requests.get( "https://api.spidra.io/api/crawl/abc-123", headers={"Authorization": "Bearer YOUR_API_KEY"} ).json() if r["status"] in ("completed", "failed", "cancelled"): break time.sleep(3) ``` ```javascript Node.js theme={null} let data; do { await new Promise(r => setTimeout(r, 3000)); const res = await fetch("https://api.spidra.io/api/crawl/abc-123", { headers: { Authorization: "Bearer YOUR_API_KEY" } }); data = await res.json(); } while (!["completed", "failed", "cancelled"].includes(data.status)); ``` ## Response Fields | Field | Type | Description | | ----------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `status` | string | `waiting`, `active`, `completed`, `failed`, or `cancelled`. | | `progress` | object | Current progress message and page counts. Present while the job is running. | | `progress.message` | string | A description of what the crawler is doing right now. | | `progress.pagesCrawled` | integer | Number of pages processed so far. | | `progress.maxPages` | integer | The page limit for this job. | | `result` | array or null | The full page result set. Populated once `status` is `completed`. `null` while the job is still running. | | `result[].url` | string | The URL of this page. | | `result[].title` | string | Page title. | | `result[].status` | string | `success` or `failed`. | | `result[].data` | any | Extracted content. Contains AI-extracted data when a `transformInstruction` or `schema` was provided. Contains the raw page markdown when neither was set. | | `result[].html` | string or null | Signed URL to the raw HTML snapshot (valid for 1 hour). | | `result[].markdown` | string or null | Signed URL to the markdown version of this page (valid for 1 hour). | | `error` | string or null | Error message if `status` is `failed`. | ## Example Responses **While running:** ```json theme={null} { "status": "active", "progress": { "message": "Crawling page 3 of 10", "pagesCrawled": 3, "maxPages": 10 }, "result": null, "error": null } ``` **Completed:** ```json theme={null} { "status": "completed", "progress": { "message": "Crawl complete", "pagesCrawled": 10, "maxPages": 10 }, "result": [ { "url": "https://example.com/blog/post-1", "title": "First Post", "status": "success", "data": { "title": "First Post", "author": "John", "date": "2025-01-01" }, "html": "https://storage.spidra.io/signed/...", "markdown": "https://storage.spidra.io/signed/..." } ], "error": null } ``` Unlike the scrape endpoint, the `result` field here is an **array** — one object per page crawled. For richer metadata on each page (page IDs, error messages, per-page timestamps), use [GET /crawl//pages](/api-reference/crawling/crawl-pages) after the job completes. # Error Handling Source: https://docs.spidra.io/api-reference/errors Spidra API error codes and response shapes — HTTP status meanings, common error messages, and how to handle retries and validation failures. Spidra returns consistent error responses using standard HTTP status codes and descriptive messages. If something goes wrong with your request—like invalid input, missing authentication, or rate limits—you’ll receive a clear JSON error object. ## Error Response Format ```json theme={null} { "status": "error", "message": "Detailed error explanation" } ``` ## Common Error Codes | Status Code | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | 400 Bad Request | The request conflicts with the current state (e.g. cancelling an already-finished job, retrying a batch with no failed items). | | 401 Unauthorized | Authentication failed — missing, invalid, or disabled API key. | | 402 Payment Required | Your subscription payment is overdue. See `PAYMENT_OVERDUE` below. | | 403 Forbidden | Your key is valid but not permitted — usually the monthly credit limit has been reached. | | 404 Not Found | The requested resource does not exist. | | 422 Unprocessable Entity | The request body failed validation. The response includes an `errors` array listing each problem. | | 429 Too Many Requests | You have exceeded a rate limit. See the `code` field for which one. | | 503 Service Unavailable | The service is temporarily at capacity. Retry after the `retry_after` value (seconds). | | 500 Internal Server Error | An unexpected error occurred on the server. | ## Validation Errors (422) All request-body validation failures on submit endpoints return `422` with an `errors` array — one entry per problem — alongside the usual `message`: ```json theme={null} { "status": "error", "message": "Request validation failed. Fix the errors below and try again.", "errors": [ "urls[0] (forEach): requires an \"observe\" field describing which elements to find", "urls[1]: not a valid URL" ] } ``` ## Error Codes Some errors include a `code` field with a machine-readable identifier: | Code | Status | Meaning | | ----------------------- | ------ | ------------------------------------------------------------------------------------------ | | `TOO_MANY_PENDING_JOBS` | 429 | You have too many jobs currently queued. Wait for some to complete before submitting more. | | `SERVICE_BUSY` | 503 | The server queue is at capacity across all users. Retry after `retry_after` seconds. | | `PAYMENT_OVERDUE` | 402 | Your payment is overdue. Update your payment method to continue. | ```json theme={null} { "status": "error", "message": "The scraping service is currently at capacity. Please retry in a few minutes.", "code": "SERVICE_BUSY", "retry_after": 60 } ``` # Introduction Source: https://docs.spidra.io/api-reference/introduction REST API reference for Spidra — scrape, batch scrape, crawl, and browser automation endpoints with full request and response examples. The Spidra API gives you programmatic access to web data. All endpoints share a common base URL, authentication scheme, and response format described on this page. ## Features Extract content from any webpage with browser automation and AI processing. Queue up to 50 URLs in one request and collect per-item results in parallel. Crawl entire websites and extract structured data from every page. ## Base URL All API requests use this base URL: ``` https://api.spidra.io/api ``` ## Authentication Include your API key in the standard `Authorization: Bearer` header. ```bash cURL theme={null} curl -X POST https://api.spidra.io/api/scrape \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": [{"url": "https://example.com"}]}' ``` ```python Python theme={null} import requests response = requests.post( "https://api.spidra.io/api/scrape", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={"urls": [{"url": "https://example.com"}]} ) ``` ```javascript Node.js theme={null} const response = await fetch("https://api.spidra.io/api/scrape", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json" }, body: JSON.stringify({ urls: [{ url: "https://example.com" }] }) }); ``` Get your API key from [app.spidra.io](https://app.spidra.io) → Settings → API Keys. Keep your API key secret. Never expose it in frontend code. ## Response Codes | Code | Meaning | What it indicates | | ----- | -------------------- | ----------------------------------------------------------------------------- | | `200` | OK | Request completed successfully | | `202` | Accepted | Job queued. Poll the status endpoint for results | | `400` | Bad Request | The request conflicts with the current state (e.g. cancelling a finished job) | | `401` | Unauthorized | API key is missing, invalid, or expired | | `402` | Payment Required | Subscription payment overdue | | `403` | Forbidden | Credits exhausted or plan limit reached | | `404` | Not Found | The job or resource does not exist | | `422` | Unprocessable Entity | The request body failed validation — the response includes an `errors` array | | `429` | Too Many Requests | Rate limit hit. Wait and retry with backoff | | `500` | Server Error | Something went wrong on our end | ## Error Format All errors return this structure: ```json theme={null} { "status": "error", "message": "Detailed error explanation" } ``` # List Scrape Logs Source: https://docs.spidra.io/api-reference/logs/scrape-logs GET /scrape-logs Retrieve your full scrape history with filtering by date, status, and URL. Access logs, token usage, and extracted results for any past job. ## Filtering Examples ```bash theme={null} # Get only successful scrapes /api/scrape-logs?status=success # Search by URL /api/scrape-logs?searchTerm=amazon.com # Date range /api/scrape-logs?dateStart=2025-12-01&dateEnd=2025-12-31 # Combine filters with pagination /api/scrape-logs?status=success&searchTerm=blog&limit=20&page=2 ``` The list endpoint excludes `result_data` for performance. Use the single-log endpoint to get full results. # Get Scrape Log Details Source: https://docs.spidra.io/api-reference/logs/scrape-logs-detail GET /scrape-logs/{uuid} Retrieve the full details of a single scrape log, including the original request, token usage, and the complete extracted result. This endpoint returns everything about a single scrape operation. Unlike the list endpoint, it includes the `result_data` field containing the full extracted content. Use this when you need to inspect or retrieve the output of a specific scrape job. ## Example Request ```bash cURL theme={null} curl https://api.spidra.io/api/scrape-logs/log-uuid-1 \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```python Python theme={null} import requests response = requests.get( "https://api.spidra.io/api/scrape-logs/log-uuid-1", headers={"Authorization": "Bearer YOUR_API_KEY"} ) ``` ```javascript Node.js theme={null} const response = await fetch( "https://api.spidra.io/api/scrape-logs/log-uuid-1", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); ``` ## Response Fields | Field | Type | Description | | ---------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------- | | `uuid` | string | Unique identifier for this log record | | `status` | string | `success`, `error`, or `in_progress` | | `started_at` | string | ISO 8601 timestamp when the scrape started | | `finished_at` | string or null | ISO 8601 timestamp when the scrape completed | | `error_message` | string or null | Details about the failure if `status` is `error` | | `urls` | array | The list of URLs that were scraped, matching the original request | | `extraction_prompt` | string or null | The AI prompt that was used, if any | | `input_tokens` | number or null | Input tokens consumed by the AI | | `output_tokens` | number or null | Output tokens generated by the AI | | `tokens_used` | number or null | Total tokens consumed | | `captcha_solved_count` | number or null | Number of CAPTCHAs solved during this request | | `scraped_sites_count` | number or null | Number of URLs successfully scraped | | `latency_ms` | number or null | Total duration of the scrape in milliseconds | | `credits_used` | number or null | Credits deducted from your account | | `result_data` | object or string or null | The full extracted content. The shape depends on whether you used a `prompt` and what `output` format was set | ## Example Response ```json theme={null} { "status": "success", "data": { "uuid": "log-uuid-1", "status": "success", "started_at": "2025-12-17T15:00:00Z", "finished_at": "2025-12-17T15:00:05Z", "error_message": null, "urls": [ { "url": "https://example.com/products/laptop" } ], "extraction_prompt": "Extract the product name, price, and availability", "input_tokens": 1240, "output_tokens": 86, "tokens_used": 1326, "captcha_solved_count": 0, "scraped_sites_count": 1, "latency_ms": 4820, "credits_used": 2, "result_data": { "product_name": "ProBook 15 Laptop", "price": 899.99, "availability": "In stock" } } } ``` The list endpoint (`GET /scrape-logs`) excludes `result_data` to keep responses fast when you are loading many records. Always use this single-record endpoint when you need the actual scraped content. # Rate Limits Source: https://docs.spidra.io/api-reference/rate-limits Spidra API rate limits — submission limits per user per minute. How limits apply, what to expect when you hit them, and retry best practices. Spidra enforces rate limits to ensure fair usage and platform stability. This page explains the limits and best practices for handling them. Submission limits are enforced per user account: **60 scrape jobs/minute** and **20 batch jobs/minute**. Polling (GET) endpoints are not rate-limited. ## Two Types of Limits Spidra enforces two separate limits. Understanding the difference matters when building integrations. ### 1. Submission Rate Limit (HTTP 429) Controls how fast you can submit new jobs — **60 scrape submissions/minute** and **20 batch submissions/minute**, both tracked per user account. Polling an existing job's status is exempt from this limit. Response: ```json theme={null} { "status": "error", "message": "Too many requests, please try again later." } ``` Includes standard headers: | Header | Description | | --------------------- | -------------------------------------- | | `RateLimit-Limit` | Maximum requests allowed in the window | | `RateLimit-Remaining` | Requests remaining in current window | | `RateLimit-Reset` | Unix timestamp when the limit resets | ### 2. Service Capacity (HTTP 503 with code `SERVICE_BUSY`) When the platform is under heavy load from all users combined, new jobs are temporarily rejected to protect stability. This is rare but can occur during traffic spikes. Response: ```json theme={null} { "status": "error", "message": "The scraping service is currently at capacity. Please retry in a few minutes.", "code": "SERVICE_BUSY", "retry_after": 60 } ``` Retry after the number of seconds in `retry_after`. ## Handling All Limits in Code ```javascript theme={null} async function submitJob(request, maxRetries = 10) { for (let attempt = 0; attempt < maxRetries; attempt++) { const response = await fetch('https://api.spidra.io/api/scrape', { method: 'POST', headers: { Authorization: 'Bearer YOUR_API_KEY', 'Content-Type': 'application/json' }, body: JSON.stringify(request) }); if (response.ok) return response.json(); const body = await response.json(); if (response.status === 503 || body.code === 'SERVICE_BUSY') { // Platform at capacity — wait and retry const wait = (body.retry_after ?? 60) * 1000; await new Promise(r => setTimeout(r, wait)); continue; } if (response.status === 429) { // HTTP rate limit — exponential backoff const wait = Math.pow(2, attempt) * 1000; await new Promise(r => setTimeout(r, wait)); continue; } throw new Error(`Request failed: ${body.message}`); } throw new Error('Max retries reached'); } ``` ### 2. Batch URLs Efficiently Instead of making multiple single-URL requests, batch up to your plan's limit: ```json theme={null} { "urls": [ {"url": "https://example.com/page1"}, {"url": "https://example.com/page2"}, {"url": "https://example.com/page3"} ], "prompt": "Extract the title and main content" } ``` ### 3. Use Crawl for Large Sites For scraping many pages from the same domain, use the [Crawl API](/api-reference/crawling/crawl) instead of multiple scrape requests. One crawl request can process up to 50 pages. ### 4. Cache Results Store scrape results locally to avoid redundant requests for the same content. Some limits vary depending on your plan. The number of concurrent URLs per scrape request and the number of actions per URL both increase on higher tiers. See the [Plans and Pricing](/billing/plans) page for a full breakdown. ## Increasing Limits Need higher limits? [Contact us](mailto:hello@spidra.io) to discuss Enterprise plans with custom rate limits and dedicated infrastructure. # Submit a Batch Scrape Job Source: https://docs.spidra.io/api-reference/scraping/batch-scrape POST /batch/scrape Queue up to 50 URLs for parallel scraping in a single request ## How It Works Batch scrape jobs are asynchronous. Submitting returns a `batchId` immediately. Each URL is processed in parallel by independent workers. 1. **Submit** — Send your URL list. Receive `batchId` in the response. 2. **Process** — Each URL is opened in a real browser, CAPTCHAs solved, content extracted. 3. **Poll** — Call `GET /api/batch/scrape/{batchId}` every 2–5 seconds until `status` is terminal. Credits are reserved upfront when you submit. The final amount is reconciled per item once processing completes. *** ## Minimal Example ```bash cURL theme={null} curl -X POST https://api.spidra.io/api/batch/scrape \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": ["https://example.com/page-1", "https://example.com/page-2"], "prompt": "Extract the headline and summary", "output": "json" }' ``` ```javascript Node.js theme={null} const res = await fetch("https://api.spidra.io/api/batch/scrape", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ urls: ["https://example.com/page-1", "https://example.com/page-2"], prompt: "Extract the headline and summary", output: "json", }), }); const { batchId, total } = await res.json(); // batchId → poll GET /api/batch/scrape/{batchId} ``` ```python Python theme={null} import requests resp = requests.post( "https://api.spidra.io/api/batch/scrape", headers={"Authorization": "Bearer YOUR_API_KEY"}, json={ "urls": ["https://example.com/page-1", "https://example.com/page-2"], "prompt": "Extract the headline and summary", "output": "json", }, ) batch_id = resp.json()["batchId"] ``` **Response `202 Accepted`:** ```json theme={null} { "status": "queued", "batchId": "f3a2b1c0-0000-0000-0000-000000000000", "total": 2 } ``` The `Location` response header is also set to `/api/batch/scrape/{batchId}` for convenience. *** ## With Structured Output Pass a `schema` to receive a consistent JSON shape for every item: ```json theme={null} { "urls": [ "https://shop.example.com/item/100", "https://shop.example.com/item/101" ], "prompt": "Extract the product details", "schema": { "type": "object", "required": ["name", "price"], "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "currency": { "type": ["string", "null"] }, "available": { "type": ["boolean", "null"] } } } } ``` When `schema` is provided, `output` is automatically forced to `"json"`. Non-fatal schema issues are returned as `schema_warnings` in the submission response. Use the [Spidra JSON Schema Generator](https://spidra.io/tools/json-schema-generator) to build and preview your schema visually before pasting it here. *** ## With Proxy ```json theme={null} { "urls": ["https://amazon.de/dp/B123", "https://amazon.de/dp/B456"], "prompt": "Extract price and availability", "output": "json", "useProxy": true, "proxyCountry": "de" } ``` *** ## With Screenshots ```json theme={null} { "urls": ["https://example.com"], "screenshot": true, "fullPageScreenshot": true } ``` Screenshot URLs are returned in each item's `screenshotUrl` field once processing is complete. *** ## Request Body | Field | Type | Required | Default | Description | | -------------------- | ------------------------ | -------- | -------- | -------------------------------------------------------------------------------------------------------------------- | | `urls` | `string[]` | Yes | — | URLs to scrape. **1–50 URLs per request.** Must be `http://` or `https://`. Private/internal IPs are rejected. | | `prompt` | `string` | No | — | AI extraction instruction applied to every URL in the batch | | `output` | `"json"` \| `"markdown"` | No | `"json"` | Output format for extracted content. Automatically `"json"` when `schema` is set | | `schema` | `object` | No | — | JSON Schema object that constrains the AI output. Validated before queuing — returns `422` if invalid | | `useProxy` | `boolean` | No | `false` | Route each URL through residential stealth proxies | | `proxyCountry` | `string` | No | — | ISO country code (`"us"`, `"de"`, `"gb"`) or region (`"eu"`, `"global"`). Requires `useProxy: true` | | `extractContentOnly` | `boolean` | No | `false` | Strip navigation, headers, and sidebars — keeps only the main content | | `cookies` | `string` | No | — | Session cookies for authenticated pages. **Never persisted to the database** — passed ephemerally to the worker only | | `screenshot` | `boolean` | No | `false` | Capture a viewport screenshot of each page | | `fullPageScreenshot` | `boolean` | No | `false` | Capture the full scrollable page. Requires `screenshot: true` | *** ## Response | Field | Type | Description | | ----------------- | ---------- | ---------------------------------------------------------------------------------------- | | `status` | `"queued"` | Always `"queued"` on a successful submission | | `batchId` | `string` | UUID — use this to poll status and manage the batch | | `total` | `number` | Number of URLs accepted into the batch | | `schema_warnings` | `string[]` | Non-fatal schema issues (e.g., unsupported keywords). Only present if there are warnings | *** ## Errors | Code | Reason | | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `401` | Missing or invalid API key | | `402` | Payment overdue — update your payment method | | `403` | Monthly credit limit reached | | `422` | Request body failed validation: `urls` missing/empty/over 50 items, invalid URLs, or malformed `schema`. An `errors` array lists each problem | | `429` | More than 20 batch submissions per minute | **Validation error example:** ```json theme={null} { "status": "error", "message": "Request validation failed. Fix the errors below and try again.", "errors": [ "URL 2: \"ftp://example.com\" is not a valid URL — must use http or https", "URL 4: private and internal URLs are not allowed" ] } ``` *** Poll for results Full feature walkthrough # Cancel a Batch Source: https://docs.spidra.io/api-reference/scraping/batch-scrape-cancel DELETE /batch/scrape/{batchId} Stop a running or pending batch and refund credits for unprocessed items ## Overview Cancels a batch that is currently `pending` or `running`. Credits reserved for items that have **not yet started** are refunded automatically. Items already being processed will complete normally. ```bash cURL theme={null} curl -X DELETE https://api.spidra.io/api/batch/scrape/YOUR_BATCH_ID \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```javascript Node.js theme={null} const res = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}`, { method: "DELETE", headers: { Authorization: "Bearer YOUR_API_KEY" }, } ); const { cancelledItems, creditsRefunded } = await res.json(); console.log(`Cancelled ${cancelledItems} items, refunded ${creditsRefunded} credits`); ``` ```python Python theme={null} import requests resp = requests.delete( f"https://api.spidra.io/api/batch/scrape/{batch_id}", headers={"Authorization": "Bearer YOUR_API_KEY"}, ) data = resp.json() print(f"Cancelled {data['cancelledItems']} items, refunded {data['creditsRefunded']} credits") ``` *** ## Response ```json theme={null} { "status": "cancelled", "cancelledItems": 8, "creditsRefunded": 16 } ``` | Field | Type | Description | | ----------------- | ------------- | ----------------------------------------------------------------------------------------- | | `status` | `"cancelled"` | Confirms the batch was cancelled | | `cancelledItems` | `number` | Number of pending items that were stopped. Items already running are not counted | | `creditsRefunded` | `number` | Credits returned to your balance. Calculated as `cancelledItems × reservedCreditsPerItem` | Items that were already `running` when you cancelled will finish processing and consume their credits. Only `pending` items are cancelled and refunded. *** ## Errors | Code | Reason | | ----- | -------------------------------------------------------------------------------------------------------- | | `401` | Missing API key authentication header | | `403` | Invalid or expired API key | | `404` | No batch found with this ID, or it belongs to a different user | | `409` | The batch is already in a terminal state (`completed`, `failed`, or `cancelled`) and cannot be cancelled | **Already terminal:** ```json theme={null} { "status": "error", "message": "Cannot cancel a batch that is already completed.", "code": "ALREADY_TERMINAL" } ``` *** Re-queue failed items instead of cancelling Check current status before cancelling # List Batch Jobs Source: https://docs.spidra.io/api-reference/scraping/batch-scrape-list GET /batch/scrape Retrieve a paginated list of all your batch scrape jobs Returns a paginated list of all batch jobs for your account, ordered by most recent first. Item-level results are not included. Use `GET /api/batch/scrape/{batchId}` to fetch those for a specific batch. ```bash cURL theme={null} curl "https://api.spidra.io/api/batch/scrape?page=1&limit=20" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```javascript Node.js theme={null} const res = await fetch( "https://api.spidra.io/api/batch/scrape?page=1&limit=20", { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); const { jobs, pagination } = await res.json(); ``` ```python Python theme={null} import requests resp = requests.get( "https://api.spidra.io/api/batch/scrape", headers={"Authorization": "Bearer YOUR_API_KEY"}, params={"page": 1, "limit": 20}, ) data = resp.json() jobs = data["jobs"] pagination = data["pagination"] ``` *** ## Query Parameters | Parameter | Type | Default | Description | | --------- | -------- | ------- | --------------------------- | | `page` | `number` | `1` | Page number (1-based) | | `limit` | `number` | `20` | Jobs per page. Maximum `50` | *** ## Response ```json theme={null} { "jobs": [ { "uuid": "f3a2b1c0-0000-0000-0000-000000000000", "status": "completed", "totalUrls": 10, "completedCount": 9, "failedCount": 1, "outputFormat": "json", "scrapingChannel": "api", "createdAt": "2024-01-15T10:00:00Z", "finishedAt": "2024-01-15T10:01:30Z" }, { "uuid": "e2c1a0b9-0000-0000-0000-000000000000", "status": "running", "totalUrls": 25, "completedCount": 12, "failedCount": 0, "outputFormat": "json", "scrapingChannel": "api", "createdAt": "2024-01-15T10:05:00Z", "finishedAt": null } ], "pagination": { "page": 1, "limit": 20, "total": 47, "totalPages": 3 } } ``` ### Job Entry Fields | Field | Type | Description | | ----------------- | ------------------ | ------------------------------------------------------------- | | `uuid` | `string` | Batch ID — use this with other endpoints | | `status` | `string` | `pending`, `running`, `completed`, `failed`, or `cancelled` | | `totalUrls` | `number` | How many URLs were in this batch | | `completedCount` | `number` | Items that completed successfully | | `failedCount` | `number` | Items that failed | | `outputFormat` | `string` | `"json"` or `"markdown"` | | `scrapingChannel` | `string` | `"api"` or `"playground"` | | `createdAt` | `string` | ISO 8601 submission timestamp | | `finishedAt` | `string` \| `null` | ISO 8601 completion timestamp, or `null` if still in progress | ### Pagination Fields | Field | Type | Description | | ------------ | -------- | ------------------------------------------- | | `page` | `number` | Current page number | | `limit` | `number` | Items per page | | `total` | `number` | Total number of batch jobs for your account | | `totalPages` | `number` | Total pages at the current `limit` | *** ## Errors | Code | Reason | | ----- | ------------------------------------- | | `401` | Missing API key authentication header | | `403` | Invalid or expired API key | *** Fetch full results for a specific batch Start a new batch job # Retry Failed Batch Scrapes Source: https://docs.spidra.io/api-reference/scraping/batch-scrape-retry POST /batch/scrape/{batchId}/retry Re-queue only the failed scrapes in a completed batch After a batch completes with some failures, retry only the failed scrapes. Successful items are never touched. Fresh credits are reserved for the retry, and the batch status resets to `running`. ```bash cURL theme={null} curl -X POST https://api.spidra.io/api/batch/scrape/YOUR_BATCH_ID/retry \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```javascript Node.js theme={null} const res = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}/retry`, { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY" }, } ); const { retriedCount } = await res.json(); console.log(`${retriedCount} items re-queued`); // Poll the same batchId until it completes again ``` ```python Python theme={null} import requests resp = requests.post( f"https://api.spidra.io/api/batch/scrape/{batch_id}/retry", headers={"Authorization": "Bearer YOUR_API_KEY"}, ) print(f"{resp.json()['retriedCount']} items re-queued") ``` *** ## Full Retry Pattern ```javascript theme={null} async function runWithRetry(urls, options, maxAttempts = 3) { // Submit the initial batch const submit = await fetch("https://api.spidra.io/api/batch/scrape", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ urls, ...options }), }); const { batchId } = await submit.json(); for (let attempt = 1; attempt <= maxAttempts; attempt++) { // Poll until terminal let data; while (true) { const res = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}`, { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); data = await res.json(); if (["completed", "failed", "cancelled"].includes(data.status)) break; await new Promise((r) => setTimeout(r, 3000)); } if (data.failedCount === 0) break; // All done if (attempt < maxAttempts) { console.log(`Attempt ${attempt}: ${data.failedCount} failed. Retrying...`); await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}/retry`, { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY" } } ); } } // Fetch final state const final = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}`, { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); return (await final.json()).items; } ``` *** ## Response **`202 Accepted`:** ```json theme={null} { "status": "queued", "retriedCount": 3 } ``` | Field | Type | Description | | -------------- | ---------- | ------------------------------------------ | | `status` | `"queued"` | Confirms the retry was accepted | | `retriedCount` | `number` | Number of failed items that were re-queued | After a successful retry: * Failed items reset to `pending` status * Batch `failedCount` decrements by `retriedCount` * Batch `status` resets to `running` * Poll the same `batchId` — it will complete again when all retried items finish *** ## Errors | Code | Reason | | ----- | -------------------------------------------------------------------------- | | `400` | No failed items to retry in this batch | | `401` | Missing API key authentication header | | `403` | Monthly credit limit reached — not enough credits to reserve for the retry | | `404` | No batch found with this ID, or it belongs to a different user | **No failed items:** ```json theme={null} { "status": "error", "message": "No failed items to retry.", "code": "NO_FAILED_ITEMS" } ``` *** Poll for results after retrying Stop the batch entirely instead # Get Batch Status Source: https://docs.spidra.io/api-reference/scraping/batch-scrape-status GET /batch/scrape/{batchId} Poll for batch progress and retrieve per-item results ## Polling Pattern Poll this endpoint every **2–5 seconds** after submitting a batch. Once `status` is `completed`, `failed`, or `cancelled`, stop polling (the batch will not change further). ```javascript Node.js theme={null} async function pollBatch(batchId) { while (true) { const res = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}`, { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); const data = await res.json(); console.log(`${data.completedCount}/${data.totalUrls} complete, ${data.failedCount} failed`); if (["completed", "failed", "cancelled"].includes(data.status)) { return data; } await new Promise((r) => setTimeout(r, 3000)); } } ``` ```python Python theme={null} import time, requests def poll_batch(batch_id): headers = {"Authorization": "Bearer YOUR_API_KEY"} while True: data = requests.get( f"https://api.spidra.io/api/batch/scrape/{batch_id}", headers=headers, ).json() print(f"{data['completedCount']}/{data['totalUrls']} complete, {data['failedCount']} failed") if data["status"] in ("completed", "failed", "cancelled"): return data time.sleep(3) ``` ```bash cURL theme={null} curl https://api.spidra.io/api/batch/scrape/YOUR_BATCH_ID \ -H "Authorization: Bearer YOUR_API_KEY" ``` *** ## Batch Status Values | Status | Meaning | | ----------- | ---------------------------------------------------------------------------- | | `pending` | Queued — no items have started yet | | `running` | At least one item is being processed | | `completed` | All items reached a terminal state. Check `failedCount` for partial failures | | `failed` | The batch failed unexpectedly | | `cancelled` | Cancelled via `DELETE /api/batch/scrape/{batchId}` | `completed` does not guarantee every URL succeeded. A batch is marked `completed` when all items have a terminal status (`completed` or `failed`). Always inspect `failedCount` and individual item statuses to detect partial failures. *** ## Response **Example — batch in progress:** ```json theme={null} { "status": "running", "totalUrls": 5, "completedCount": 3, "failedCount": 0, "items": [...], "createdAt": "2024-01-15T10:00:00Z", "finishedAt": null } ``` **Example — batch completed:** ```json theme={null} { "status": "completed", "totalUrls": 5, "completedCount": 4, "failedCount": 1, "items": [...], "createdAt": "2024-01-15T10:00:00Z", "finishedAt": "2024-01-15T10:00:42Z" } ``` ### Top-Level Fields | Field | Type | Description | | ---------------- | ------------------ | -------------------------------------------------------------------------------------- | | `status` | `string` | Batch-level status: `pending`, `running`, `completed`, `failed`, or `cancelled` | | `totalUrls` | `number` | Total number of URLs in the batch | | `completedCount` | `number` | Items that finished successfully | | `failedCount` | `number` | Items that encountered an error | | `items` | `array` | Per-item results (see below) | | `createdAt` | `string` | ISO 8601 timestamp when the batch was submitted | | `finishedAt` | `string` \| `null` | ISO 8601 timestamp when the batch reached a terminal state, or `null` if still running | *** ## Per-Item Fields Each entry in `items` represents one URL: ```json theme={null} { "uuid": "a1b2c3d4-0000-0000-0000-000000000000", "url": "https://example.com/product/42", "jobId": "bull-worker-job-id", "status": "completed", "result": { "name": "Widget Pro", "price": 49.99, "available": true }, "error": null, "creditsUsed": 3, "startedAt": "2024-01-15T10:00:05Z", "finishedAt": "2024-01-15T10:00:11Z", "screenshotUrl": null } ``` | Field | Type | Description | | --------------- | ------------------ | --------------------------------------------------------------------------------------------- | | `uuid` | `string` | Unique ID for this batch item | | `url` | `string` | The URL that was processed | | `jobId` | `string` \| `null` | Internal worker job ID. `null` if the item is still pending | | `status` | `string` | Item-level status: `pending`, `running`, `completed`, or `failed` | | `result` | `any` \| `null` | Extracted content. Object if `output: "json"`, string if `"markdown"`. `null` until completed | | `error` | `string` \| `null` | Error description if `status` is `failed`, otherwise `null` | | `creditsUsed` | `number` | Credits consumed by this item. `0` for failed or cancelled items | | `startedAt` | `string` \| `null` | ISO 8601 timestamp when the worker started this item | | `finishedAt` | `string` \| `null` | ISO 8601 timestamp when this item completed or failed | | `screenshotUrl` | `string` \| `null` | S3 URL for the screenshot, or `null` if screenshots were not requested | *** ## Handling Partial Failures When `completedCount + failedCount === totalUrls` but some items failed, retry them without re-running the whole batch: ```javascript theme={null} const data = await pollBatch(batchId); if (data.failedCount > 0) { const retry = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}/retry`, { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY" } } ); const { retriedCount } = await retry.json(); console.log(`Retrying ${retriedCount} failed items...`); } ``` *** ## Errors | Code | Reason | | ----- | -------------------------------------------------------------- | | `401` | Missing API key authentication header | | `403` | Invalid or expired API key | | `404` | No batch found with this ID, or it belongs to a different user | *** Re-queue only the failed items Stop processing and refund credits # Submit a Scrape Job Source: https://docs.spidra.io/api-reference/scraping/scrape POST /scrape Queue URLs for scraping with optional browser actions and AI extraction ## How It Works Spidra runs scrape jobs asynchronously. When you submit a request, you get a `jobId` back immediately. You then poll `GET /scrape/{jobId}` until `status` is `completed` and results are ready. 1. **Submit** - Send your request, receive a `jobId` in the response right away 2. **Load** - Spidra opens each URL in a real browser 3. **Execute** - Runs your browser actions (clicks, scrolls, etc.) 4. **Solve** - Automatically handles CAPTCHAs 5. **Process** - Runs AI extraction if a `prompt` is provided 6. **Poll** - Check `GET /scrape/{jobId}` until `status: "completed"` Setting `output: "json"` without a `prompt` still triggers a default AI extraction pass. If you want raw markdown with no AI processing, omit both `output` and `prompt`. When AI extraction fails (for example, on a near-empty page), Spidra falls back to returning the raw page markdown in `result.content`. Check the `ai_extraction_failed` flag in the response to detect this case and handle degraded results in your code. *** ## Structured Output Pass a `schema` to tell the AI exactly what shape to return. Instead of getting whatever JSON the AI decides to produce, you get back a JSON object that matches your schema every time. Nullable fields come back as `null` rather than being omitted. Field names match exactly what you defined. ```json theme={null} { "urls": [{ "url": "https://jobs.example.com/engineer" }], "prompt": "Extract the job details. Normalize salary to a plain number in USD.", "schema": { "type": "object", "required": ["title", "company", "remote", "employment_type"], "properties": { "title": { "type": "string" }, "company": { "type": "string" }, "remote": { "type": ["boolean", "null"] }, "salary_min": { "type": ["number", "null"] }, "salary_max": { "type": ["number", "null"] }, "employment_type": { "type": ["string", "null"], "enum": ["full_time", "part_time", "contract", null] } } } } ``` `output` is automatically set to `"json"` when a schema is provided. The schema is validated before the job is queued and a `422` is returned with descriptive errors if the schema is malformed. Non-fatal issues (unsupported keywords) are returned as `schema_warnings` in the job status response. Full guide: nested objects, arrays, nullable fields, the required rule, and schema limits Generate your schema from an existing Zod or Pydantic model Build and preview your schema visually — no JSON knowledge required *** ## Browser Actions Interact with the page before scraping — click buttons, fill forms, scroll to load content, dismiss modals, and iterate over lists of elements. ```json theme={null} { "urls": [{ "url": "https://example.com/products", "actions": [ {"type": "click", "selector": "#accept-cookies"}, {"type": "wait", "duration": 1500}, {"type": "scroll", "to": "80%"} ] }], "prompt": "List all product names and prices", "output": "json" } ``` ### Available Actions | Action | What it does | Quick example | | --------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | `click` | Clicks any element on the page, buttons, links, tabs, toggles | `{"type": "click", "selector": "#load-more"}` | | `type` | Types text into an input field or search box | `{"type": "type", "selector": "#search", "value": "laptops"}` | | `check` | Checks a checkbox | `{"type": "check", "selector": "#in-stock-only"}` | | `uncheck` | Unchecks a checkbox | `{"type": "uncheck", "selector": "#newsletter"}` | | `wait` | Pauses for a number of milliseconds | `{"type": "wait", "duration": 2000}` | | `scroll` | Scrolls the page to a percentage of its height | `{"type": "scroll", "to": "80%"}` | | `forEach` | Finds all matching elements and processes each one individually. Supports navigate, click, and inline modes. | `{"type": "forEach", "observe": "Find all product cards", "mode": "navigate"}` | For `click`, `check`, and `uncheck` you can target elements by CSS selector like `"selector": "#id"` or by a plain English description like `"value": "Accept cookies button"`. Both work. `forEach` is the most powerful action. It finds a set of repeating elements (product cards, links, accordion rows) and runs a mini-scrape on each one. It supports three modes (`click`, `inline`, `navigate`), automatic pagination, per-item AI extraction, and per-element sub-actions. Detailed explanations for every action with real examples, all forEach options, chaining patterns, and sample responses *** ## Proxy and Geo-Targeting Route requests through residential proxies to avoid detection or access geo-restricted content. Set `"useProxy": true` and optionally add `"proxyCountry"` to target a specific location. ```json theme={null} { "urls": [{"url": "https://amazon.de/dp/B123456"}], "prompt": "Extract the product price in euros", "output": "json", "useProxy": true, "proxyCountry": "de" } ``` Full guide: country list, EU rotation, examples, and credit costs *** ## Extract Content Only Remove navigation, headers, footers, and sidebars before processing. Useful when you only want the main article or product content. ```json theme={null} { "urls": [{"url": "https://blog.example.com/article"}], "prompt": "Summarize this article", "output": "json", "extractContentOnly": true } ``` *** ## Screenshots Capture screenshots of scraped pages for debugging or archival. ```json theme={null} { "urls": [{"url": "https://example.com"}], "prompt": "Extract page title", "screenshot": true, "fullPageScreenshot": true } ``` | Option | Description | | -------------------------- | ---------------------------------------------------------------- | | `screenshot: true` | Capture the visible viewport | | `fullPageScreenshot: true` | Capture the entire scrollable page (requires `screenshot: true`) | Screenshot URLs are returned in the `screenshots` array of the response. *** ## Authentication Scrape protected pages by providing session cookies: ```json theme={null} { "urls": [{"url": "https://example.com/dashboard"}], "prompt": "Extract account details", "output": "json", "cookies": "session=eyJ...; auth_token=abc123..." } ``` Full guide on getting cookies and formats *** Poll for results See your scrape history # Get Scrape Job Status Source: https://docs.spidra.io/api-reference/scraping/scrape-status GET /scrape/{jobId} Poll the status of a running scrape job. Returns progress, token usage, CAPTCHA solve count, and the full extracted result when the job completes. ## Polling Pattern Scrape jobs are processed asynchronously. When you submit a job you get a `jobId` back immediately. You then poll this endpoint every 2-5 seconds until `status` is `completed` or `failed`. ```javascript theme={null} async function waitForResult(jobId) { while (true) { const res = await fetch(`https://api.spidra.io/api/scrape/${jobId}`, { headers: { Authorization: 'Bearer YOUR_API_KEY' } }); const data = await res.json(); if (data.status === 'completed') return data.result; if (data.status === 'failed') throw new Error(data.error); await new Promise(r => setTimeout(r, 3000)); } } ``` *** ## Status Values | Status | Meaning | | ----------- | ----------------------------------- | | `waiting` | In queue, not started yet | | `active` | Running right now | | `completed` | Done, results are ready | | `failed` | Something went wrong, check `error` | *** ## Response Structure When `status` is `completed`, everything you need is inside `result`. ```json theme={null} { "status": "completed", "progress": { "message": "Scrape completed successfully", "progress": 1 }, "result": { "content": "...", "screenshots": [], "ai_extraction_failed": false, "stats": { "durationMs": 4200, "captchaSolvedCount": 0, "inputTokens": 312, "outputTokens": 84, "totalTokens": 396 } }, "error": null } ``` ### result.content This is the main output field. What it contains depends on whether you provided a `prompt`: * **With `prompt`**: the AI-extracted result, formatted according to `output` (`"markdown"` or `"json"`) * **Without `prompt`**: the raw scraped page content as markdown If AI extraction fails for any reason, `content` still returns the raw markdown as a fallback, and `ai_extraction_failed` is set to `true` so you can detect this. ### result.stats Timing and usage information for the job. | Field | Description | | -------------------- | ------------------------------------------------- | | `durationMs` | How long the whole job took in milliseconds | | `captchaSolvedCount` | Number of CAPTCHAs that were automatically solved | | `inputTokens` | Tokens sent to the AI model | | `outputTokens` | Tokens returned from the AI model | | `totalTokens` | Total tokens used (input + output) | *** ## Failed Jobs When `status` is `failed`, the `error` field contains the reason: ```json theme={null} { "status": "failed", "error": "Failed to scrape https://example.com — net::ERR_NAME_NOT_RESOLVED" } ``` # Managing Your Subscription Source: https://docs.spidra.io/billing/managing-subscription Learn how to upgrade, downgrade, cancel, and reactivate your Spidra subscription. You manage your Spidra subscription from the [Billing page](https://app.spidra.io/dashboard/billing) in your dashboard. This page explains what happens when you upgrade, downgrade, cancel, or run into a payment issue. Spidra Billing page showing subscription management options ## Upgrading Your Plan To upgrade, go to the Billing page, find the plan you want, and click the upgrade button. Spidra redirects you to a Stripe checkout page where you complete the payment. Once the payment goes through: * Your account switches to the new plan immediately. * If you are upgrading from the Free plan, your credit counter resets and you start fresh on the new plan's monthly allowance. ## Downgrading Your Plan Downgrading works like cancellation. You keep your current plan and credits until the end of your billing cycle. At renewal, your account moves to the lower plan. If you downgrade to Free, your paid plan features stay active until the cycle ends. After that, your credit limit drops to the Free tier's 300 credits. ## Canceling Your Subscription To cancel, click the cancel button on the Billing page and confirm. Canceling does not cut off your access right away. Spidra sets your subscription to cancel at the end of your current billing cycle. Until that date: * You keep full access to your current plan's features. * Your remaining credits are still available to use. * You can still run scrapes and crawls. When the billing cycle ends, your account reverts to the Free plan. ### Reactivating Before Cancellation Takes Effect If you cancel and then change your mind, you can reactivate your subscription from the Billing page before the billing cycle ends. Your subscription picks up as normal, no data is lost, and you are not charged again until the next renewal date. ## What Happens When a Payment Fails If Stripe cannot charge your card, your subscription moves to **Past Due**. Stripe retries the payment automatically over the following days. During this time, your account keeps working normally. You do not lose access while Stripe is retrying. If all retries fail, your subscription ends and your account reverts to the Free plan. To avoid this, update your payment method as soon as you see the Past Due status. ### Updating Your Payment Method Click **Manage payment method** on the Billing page. This opens the Stripe Customer Portal where you can update your card details. ## Billing Cycle Renewals Your billing cycle starts on the date you first subscribed. Each month on that date, Stripe charges your card and Spidra resets your credit usage counter for the new cycle. If you have unused top-up credits, those carry forward into the new cycle. They are not affected by the monthly reset. See [Credit Top-ups](/billing/top-ups) for more on how that works. If you have any questions, [contact support](mailto:hello@spidra.io). # Plans and Pricing Source: https://docs.spidra.io/billing/plans Spidra pricing plans — Free, Starter ($19/mo), Builder ($79/mo), Pro ($249/mo), and Enterprise. Credit allowances, concurrency limits, and bandwidth quotas per plan. Spidra has four standard plans and an Enterprise option for teams with custom needs. Every plan gives you access to the core scraping and crawling features. What changes between plans is how many credits you get per month, how much proxy bandwidth is included, how many URLs you can submit per request, and what support you can expect. Spidra plans and pricing table Credits reset at the start of every billing cycle. Unused monthly credits do not carry forward. Top-up credits are the exception — see [Credit Top-ups](/billing/top-ups) for details. ## What Each Plan Is Good For * **Free** is for trying Spidra out. You get 300 credits and 50 MB of proxy bandwidth per month, which covers light testing or personal one-off scrapes. **There is no credit card required to sign up**. * **Starter** works well for small projects, side projects, or individuals who scrape regularly but not at high volume. At 5,000 credits and 500 MB of bandwidth per month you can run a few dozen scrapes daily without hitting your limit. * **Builder** is for developers building scraping into a product or workflow. 25,000 credits and 2 GB of bandwidth per month gives you a lot of room, and Advanced stealth mode means better results on sites with aggressive anti-bot protection. * **Pro** is for teams and high-volume use cases. 125,000 credits and 5 GB of proxy bandwidth per month, 20 actions per URL, and priority support are the main differences from Builder. * **Enterprise** is for organizations that need custom credit volumes, dedicated infrastructure, SLAs, or anything that does not fit the standard plans. [Contact us](mailto:hello@spidra.io) to talk through what you need. ## Switching Plans You can upgrade or downgrade from the [Billing page](https://app.spidra.io/dashboard/billing) in your dashboard at any time. When you upgrade, you move to the new plan immediately. When you downgrade or cancel, you keep your current plan until the end of your billing cycle. See [Managing Your Subscription](/billing/managing-subscription) for the full details. # Proxy Top-up Source: https://docs.spidra.io/billing/proxy-topup Buy additional proxy bandwidth on top of your monthly plan. Proxy top-ups let you buy extra proxy bandwidth on top of what your plan includes. They are a one-time purchase, not a subscription charge. If you run large crawl or scrape jobs with [Stealth Mode](/features/stealth-mode) enabled, expect a spike in proxy usage, or simply want a buffer, a proxy top-up is the quickest way to add capacity without changing your plan. Spidra proxy bandwidth top-up packages and pricing You can buy a top-up from the [Billing page](https://app.spidra.io/dashboard/billing) in your dashboard. Click the package you want, complete the Stripe checkout, and the bandwidth is added to your account immediately. ## Available Packages | Package | Bandwidth | Price | | ------- | --------- | ----- | | 1 GB | 1,000 MB | \$7 | | 5 GB | 5,000 MB | \$28 | | 10 GB | 10,000 MB | \$52 | ## How Proxy Top-ups Work Top-up bandwidth is separate from your monthly plan allowance. When you run a proxy-enabled job, Spidra always draws from your monthly bandwidth first. Top-up bandwidth only kicks in when your monthly balance hits zero. This means a top-up does not replace your plan. It acts as a reserve that covers any proxy usage that goes beyond your monthly allowance. Proxy top-ups are only available on Starter, Builder, and Pro plans. Free plan users need to upgrade to a paid plan before they can purchase additional bandwidth. ## Expiry Proxy top-up bandwidth expires 12 months after the date of purchase. Spidra sends you two email reminders before your bandwidth expires: * A reminder 30 days before the expiry date. * A second reminder 7 days before the expiry date. If you have not used the bandwidth by the expiry date, it is removed from your account automatically. Proxy top-ups are non-refundable once purchased. Make sure you will use them within 12 months before buying a large package. ## Carrying Forward Bandwidth When your billing cycle renews, any remaining top-up bandwidth carries forward to the new cycle. The monthly reset only affects your plan's included bandwidth, not top-ups. So if you buy a 1 GB top-up and only use 200 MB this month, 800 MB will still be in your account when the next cycle starts (assuming it has not expired). A plan upgrade makes more sense if you are consistently hitting your monthly bandwidth limit. Learn how proxy bandwidth is consumed Buy additional scrape credits # Credit Top-ups Source: https://docs.spidra.io/billing/top-ups Buy additional Spidra credits on top of your monthly plan. Top-ups let you buy extra credits on top of what your plan includes. They are a one-time purchase, not a subscription charge. If you are running a large scrape project, expect a spike in usage, or just want a buffer, a top-up is the quickest way to add capacity without changing your plan. Spidra credit top-up packages and pricing table You can buy a top-up from the [Billing page](https://app.spidra.io/dashboard/billing) in your dashboard. Click the package you want, complete the Stripe checkout, and the credits are added to your account immediately. ## How Top-up Credits Work Top-up credits are separate from your monthly plan credits. When you run a scrape, Spidra always draws from your monthly credits first. Top-up credits only kick in when your monthly balance hits zero. This means a top-up does not replace your plan. It acts as a reserve that covers any usage that goes beyond your monthly allowance. Top-ups are only available on Starter, Builder, and Pro plans. Free plan users need to upgrade to a paid plan before they can purchase additional credits. ## Credit Expiry Top-up credits expire 12 months after the date of purchase. Spidra sends you two email reminders before your credits expire: * A reminder 30 days before the expiry date. * A second reminder 7 days before the expiry date. If you have not used the credits by the expiry date, they are removed from your account automatically. Top-up credits are non-refundable once purchased. Make sure you will use them within 12 months before buying a large package. ## Carrying Forward Credits When your billing cycle renews, any remaining top-up credits carry forward to the new cycle. The monthly reset only affects your plan's included credits, not top-ups. So if you buy a 1,200-credit top-up and only use 200 of them this month, 1,000 credits will still be in your account when the next cycle starts (assuming they have not expired). A plan upgrade makes more sense if you are consistently hitting your monthly credit limit. # Usage and Analytics Source: https://docs.spidra.io/billing/usage-analytics Track Spidra credit consumption, request volume, token usage, and bandwidth by day, week, or month from the Usage dashboard. Spidra tracks your usage automatically and gives you a visual breakdown right in your dashboard. You can see how many credits you have spent, how many scrapes you have run, how many CAPTCHAs were solved, and more, all filterable by time range. Spidra usage analytics chart If you want to work with usage data programmatically, for example to build an internal dashboard or set up custom alerts, use the [Usage Stats API endpoint](/api-reference/account/usage-stats). ```bash theme={null} GET https://api.spidra.io/api/usage-stats?range=7d ``` The endpoint accepts `7d`, `30d`, or `weekly` as the `range` parameter and returns the same data the dashboard chart displays, including credits, requests, crawls, tokens, CAPTCHAs, and latency. See the [Usage Stats API reference](/api-reference/account/usage-stats) for the full response schema and authentication details. This helps you stay on top of your spending, catch unexpected spikes, and decide when you need more credits or a higher plan. ## Time Ranges You can filter the chart by three time ranges: | Range | What it shows | | :---------- | :--------------------------------------- | | **7 days** | Daily usage for the past 7 days | | **30 days** | Daily usage for the past 30 days | | **Weekly** | Week-by-week totals for the past 7 weeks | Click the range selector above the chart to switch between views. *** ## What the Chart Tracks The usage chart covers six metrics. You can toggle each one on or off depending on what you want to focus on. | Metric | What it means | | :------------------- | :--------------------------------------------------------- | | **Credits consumed** | Total credits spent in the selected period | | **Requests** | Number of scrape jobs submitted | | **Crawls** | Number of crawl jobs submitted | | **Tokens used** | Raw token count from AI extraction (input + output) | | **CAPTCHAs solved** | Number of CAPTCHAs Spidra bypassed on your behalf | | **Avg latency** | Average response time in milliseconds across your requests | *** ## Your Credit Balance At the top of the Billing page you will see your current credit balance as a progress bar. It shows: * How many credits you have used this billing cycle. * How many you have remaining. * Your total monthly allowance from your plan. * Any top-up credits you have available. Spidra credit usage progress bar # GitHub Source: https://docs.spidra.io/development # Browser Actions Source: https://docs.spidra.io/features/actions Automate clicks, scrolls, form fills, and element loops before scraping. ## What Are Browser Actions? When Spidra opens a page, it does not just grab the raw HTML and leave. You can tell it to interact with the page first. That means clicking buttons, filling in search boxes, scrolling down to load more content, dismissing cookie banners, or looping through every card, accordion, or link on the page. Actions run in the order you provide them, one after the other. Spidra uses a real browser, so anything a human could do on the page, your action pipeline can do too. You add actions in the `actions` array on each URL object: ```json theme={null} { "urls": [{ "url": "https://example.com/products", "actions": [ { "type": "click", "selector": "#accept-cookies" }, { "type": "wait", "duration": 1500 }, { "type": "scroll", "to": "80%" } ] }], "prompt": "List all products and their prices" } ``` *** ## How to Target Elements Most actions need to know which element on the page to interact with. There are two ways to point Spidra at an element. ### CSS or XPath selectors (the `selector` field) If you know the page structure, CSS selectors are the most precise and reliable way to target an element. ```json theme={null} { "type": "click", "selector": "#accept-cookies" } { "type": "click", "selector": ".load-more-btn" } { "type": "click", "selector": "button[data-testid='submit']" } { "type": "type", "selector": "input[name='q']", "value": "laptop" } ``` XPath works too: ```json theme={null} { "type": "click", "selector": "//button[contains(text(), 'Accept')]" } ``` ### Plain English descriptions (the `value` field) You can also describe the element in plain English and Spidra will find it on the page for you. This is useful when CSS selectors are fragile or change between page loads. ```json theme={null} { "type": "click", "value": "Accept cookies button" } { "type": "click", "value": "Load more products button at the bottom of the page" } { "type": "check", "value": "Subscribe to newsletter checkbox" } ``` For `click`, `check`, and `uncheck` you can use either `selector` (CSS or XPath) or `value` (plain English description). For `type`, use `selector` to point at the input field and `value` for the text you want to type. *** ## Actions Reference ### click Clicks any element on the page. This works for buttons, links, tabs, dropdowns, toggles, and anything else that responds to a click. Either `selector` or `value` is required. ```json theme={null} { "type": "click", "selector": "#accept-cookies" } { "type": "click", "value": "Close the cookie consent modal" } { "type": "click", "selector": "//button[text()='Load More']" } ``` Common uses: * Dismiss cookie consent banners before the real content loads * Click "Load More" to reveal additional results * Open dropdown menus or select tabs * Navigate to the next page *** ### type Types text into an input field, textarea, or search box. Both `selector` and `value` are required. ```json theme={null} { "type": "type", "selector": "input[name='q']", "value": "best laptops 2024" } { "type": "type", "selector": "#email", "value": "test@example.com" } { "type": "type", "selector": "input[placeholder='Search...']", "value": "running shoes" } ``` A common pattern is to type a query, click submit, then wait for results: ```json theme={null} { "actions": [ { "type": "type", "selector": "#search-input", "value": "running shoes" }, { "type": "click", "selector": "button[type='submit']" }, { "type": "wait", "duration": 2000 } ] } ``` *** ### check Checks a checkbox. If the checkbox is already checked, nothing happens. ```json theme={null} { "type": "check", "selector": "#agree-terms" } { "type": "check", "value": "In Stock filter checkbox" } ``` *** ### uncheck Unchecks a checkbox. If the checkbox is already unchecked, nothing happens. ```json theme={null} { "type": "uncheck", "selector": "#newsletter" } { "type": "uncheck", "value": "Show out-of-stock items checkbox" } ``` *** ### wait Pauses the scrape for a set number of milliseconds. Use this after actions that trigger loading, animations, or data fetching. The `duration` field sets the wait time in milliseconds. ```json theme={null} { "type": "wait", "duration": 2000 } { "type": "wait", "duration": 500 } ``` Add a wait after clicking "Load More" or scrolling to the bottom. This gives the browser time to fetch and render the new content before Spidra captures the page. The older `value` field (e.g. `"value": 2000`) is still accepted but deprecated. Use `duration` going forward. *** ### scroll Scrolls the page to a percentage of its total height. This is essential for pages that load content as you scroll (infinite scroll, lazy-loaded images). The `to` field takes a number or string percentage between 0 and 100. ```json theme={null} { "type": "scroll", "to": "50%" } { "type": "scroll", "to": 80 } { "type": "scroll", "to": "100%" } ``` A pattern for pages with lazy-loaded content: ```json theme={null} { "actions": [ { "type": "scroll", "to": "50%" }, { "type": "wait", "duration": 1500 }, { "type": "scroll", "to": "100%" }, { "type": "wait", "duration": 1500 } ] } ``` The older `value` field (e.g. `"value": "80%"`) is still accepted but deprecated. Use `to` going forward. *** ## forEach: Process Every Element on a Page `forEach` is the most powerful action in Spidra. Rather than scraping a page once and hoping all the data is there, `forEach` finds a set of matching elements on the page and processes each one individually. It then combines all the results into one output. Think of it as running a mini scrape on every item in a list. ### Adding forEach `forEach` is an action, so it goes inside the `actions` array along with any other actions you want to run first. Use the `observe` field to describe which elements to find. ```json theme={null} { "urls": [{ "url": "https://example.com/products", "actions": [ { "type": "click", "selector": "#filter-electronics" }, { "type": "forEach", "observe": "Find all product cards", "mode": "inline", "maxItems": 20, "itemPrompt": "Extract product name and price. Return as JSON: {name, price}" } ] }] } ``` Any actions listed before the `forEach` run once on the page first. The `forEach` then runs on whatever state the page is in after those actions complete. ### Writing a good observe instruction The `observe` field is the most important part of forEach. It tells Spidra which elements to find on the page. Vague instructions produce inconsistent results. **Describe what the elements are, not what you want to do with them:** ``` Good: "Find all book cards in the product grid" Good: "Find all quote blocks on the page" Good: "Find all room type cards" Too vague: "Find items" Action-oriented (avoid): "Click every product" ``` **Be specific about location and type:** ``` Good: "Find all article elements with class product_pod" Good: "Find all list items in the search results container" Good: "Find all anchor tags inside the product grid" ``` **In inline mode with a CSS `captureSelector`:** the `observe` instruction helps Spidra locate the set of elements to iterate over, but the `captureSelector` CSS is what actually reads each element's content. Keep them consistent. If `observe` says "product cards" then `captureSelector` should point at the same element (e.g. `article.product_pod`), not a child element. *** ### Do you actually need forEach? Before reaching for forEach, consider whether a top-level `prompt` is enough. **If the list is short and fits on one page, you usually do not need forEach at all.** Just scrape the URL and use `prompt` to extract what you need from the full page. It is simpler and works just as well. ```json theme={null} { "urls": [{ "url": "https://quotes.toscrape.com" }], "prompt": "List all quotes and their authors" } ``` **Use forEach when:** * You need to collect items across multiple pages using `pagination`. A top-level prompt only sees the page it lands on. It cannot follow next-page links on its own. * You have 20 or more items and want `itemPrompt` to extract fields from each one individually, keeping each AI call small and the output consistently structured. * You are using `navigate` or `click` mode to access content that is only available after clicking into each item. *** ### The three forEach modes The `mode` field tells Spidra how to interact with each element it finds. #### inline mode: Read the element directly Use this when the data is visible on the page inside each matched element, and you do not need to click anything. Product cards, quote blocks, search result rows, table rows. Spidra reads each element's content and moves on. The page is not changed between items. ```json theme={null} { "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 10, "itemPrompt": "Extract title, price, and star rating. Return as JSON: {title, price, star_rating}" }] } ``` Sample response: ``` ## Item 1 {"title": "Sharp Objects", "price": "£47.82", "star_rating": "Four"} --- ## Item 2 {"title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One"} --- ## Item 3 {"title": "When We Collided", "price": "£31.77", "star_rating": "One"} ``` For a short single-page list, `inline` mode without `pagination` or `itemPrompt` gives you structured output, but a top-level `prompt` on the raw page produces equivalent results with less setup. The main reasons to use inline are pagination across pages and per-item AI extraction at scale (20+ items). *** #### navigate mode: Follow each link to its destination page Use this when each element is a link and the content you want lives on the page it points to. Product listings, search results, category pages where clicking a card takes you to the full detail page. Spidra clicks each element, loads the destination page, captures it, then returns and moves to the next element. ```json theme={null} { "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 6, "waitAfterClick": 800, "itemPrompt": "Extract title, price, star rating (One through Five), and availability. Return as JSON." }] } ``` Sample response: ``` ## Item 1 { "title": "Sharp Objects", "price": "£47.82", "star_rating": "Four", "availability": "In stock" } --- ## Item 2 { "title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One", "availability": "In stock" } ``` Navigate mode is slower than inline because it loads a full page per item. But it gives you access to all the rich detail that only exists on the individual item page, like full descriptions, specifications, reviews, and related content. *** #### click mode: Click to expand, capture, then move on Use this for pages where clicking an element opens more content within the same page. Hotel room cards that open modals, FAQ rows that expand, product variant selectors that reveal details. Spidra clicks each element, waits for the expanded content to appear, captures it, then closes and moves to the next. ```json theme={null} { "url": "https://hotels.example.com/hotel/grand-plaza", "actions": [{ "type": "forEach", "observe": "Find all room category cards", "mode": "click", "captureSelector": "[role='dialog']", "waitAfterClick": 1200, "itemPrompt": "Extract room name, bed type, price per night, and amenities. Return as JSON." }] } ``` Sample response: ``` ## Item 1 { "room": "Deluxe King Room", "bed_type": "1 King Bed", "price_per_night": "$189", "amenities": ["Free WiFi", "City view", "Air conditioning", "Mini bar"] } --- ## Item 2 { "room": "Standard Twin Room", "bed_type": "2 Twin Beds", "price_per_night": "$129", "amenities": ["Free WiFi", "Garden view", "Air conditioning"] } ``` If no `captureSelector` is provided, Spidra first tries to find an open modal (`[role="dialog"]`) and falls back to capturing the full page. *** ### captureSelector This tells Spidra which part of the page to read for each item. Without it, Spidra captures whatever is most relevant by default, but being specific gives you cleaner, more focused results. You can use a CSS selector or a plain English description: ```json theme={null} "captureSelector": "article.product_pod" "captureSelector": "article.product_page" "captureSelector": "[role='dialog']" "captureSelector": "The pricing table in the center of the page" ``` If you leave it out: * In `click` mode: Spidra looks for an open modal first, then captures the full page * In `navigate` mode: Spidra captures the full destination page * In `inline` mode: Spidra captures the full HTML of each matched element CSS selectors are more reliable for `captureSelector`. When you use plain English, Spidra tries to locate the element using AI, but if it cannot find a match it falls back to capturing the full page content instead. For consistent production results, use a CSS selector when you know the structure of the page. *** ### itemPrompt: Extract specific fields from each item `itemPrompt` runs an AI extraction on each item individually, right after it is captured. You tell it exactly what fields to pull out and in what format. This is different from the top-level `prompt`, which runs once on the full combined output after all items are collected. Why use itemPrompt: * Each item is processed on its own, so the AI has full focus on just that one item * Very useful in navigate mode where each destination page has a lot of unrelated content * Keeps output clean and structured per item from the start * The top-level prompt still runs afterwards if you also provide one ```json theme={null} "itemPrompt": "Extract the book title and price. Return as JSON: {title, price}" "itemPrompt": "Extract title, price in £XX.XX format, star rating as a word (One through Five), and availability. Return as JSON." "itemPrompt": "Get the room name, nightly rate, and list of included amenities. Return as JSON." ``` Without `itemPrompt`, each item's raw captured content is returned as markdown text, and your top-level `prompt` handles all the extraction at the end. *** ### pagination: Keep going to the next page After processing all elements on the current page, forEach can follow the next-page link and continue collecting until it hits your limit or runs out of pages. You need to tell it which button or link goes to the next page: ```json theme={null} { "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book title links", "mode": "navigate", "maxItems": 20, "itemPrompt": "Extract title, price, and rating as JSON", "pagination": { "nextSelector": "li.next > a", "maxPages": 3 } }] } ``` How it works: once all items on the first page are collected, Spidra clicks the next page button, waits for the new items to load, and continues. It stops when it has collected `maxItems` total across all pages, when it has visited `maxPages` additional pages, or when there is no next page button left. `nextSelector` examples: ```json theme={null} "nextSelector": "li.next > a" "nextSelector": "a[rel='next']" "nextSelector": ".pagination .next" "nextSelector": "The Next page button at the bottom right" ``` `maxPages` is the number of extra pages beyond the first one. Setting `maxPages: 3` means Spidra processes the starting page plus 3 more, so 4 pages total. *** ### Per-element actions: Do something on each item before capturing The `actions` field inside forEach lets you run browser actions after landing on each item but before capturing its content. This is useful when the destination page needs a scroll to load the full content, an extra click to expand a section, or a moment to settle before reading. ```json theme={null} { "url": "https://books.toscrape.com", "actions": [ { "type": "click", "selector": "a[href='catalogue/category/books/poetry_23/index.html']" }, { "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 3, "waitAfterClick": 1000, "actions": [ { "type": "scroll", "to": "50%" } ], "itemPrompt": "Extract title, price, star rating, and the full product description. Return as JSON: {title, price, star_rating, description}" } ] } ``` Sample response: ``` ## Item 1 { "title": "Poetry Unbound: 50 Poems to Open Your World", "price": "£23.00", "star_rating": "Five", "description": "Selected and introduced by Padraig O Tuama, this anthology brings together..." } ``` *** ### maxItems and waitAfterClick **`maxItems`** sets a cap on how many elements to process. The default is 50 and the maximum allowed is also 50. This limit applies across all pages combined when you use pagination. ```json theme={null} "maxItems": 10 "maxItems": 50 ``` **`waitAfterClick`** is how long to wait in milliseconds after clicking or navigating before capturing the content. The default is 2500ms. You can lower this for fast static pages or raise it for pages that fetch content from an API after load. ```json theme={null} "waitAfterClick": 500 // Static pages "waitAfterClick": 1200 // Pages with animations "waitAfterClick": 3000 // Slow or API-driven content ``` *** ## Advanced Patterns ### Pattern 1: Click to a category first, then forEach over its items The pre-actions run once when the URL loads. The forEach runs on whatever page the browser is on after those actions finish. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com", "actions": [ { "type": "click", "selector": "a[href='catalogue/category/books/travel_2/index.html']" }, { "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 4, "waitAfterClick": 800, "itemPrompt": "Extract the book title and price. Return as JSON: {title, price}" } ] }], "output": "json" } ``` Result, 4 Travel books with title and price: ```json theme={null} { "result": { "content": "## Item 1\n\n{\"title\": \"It's Only the Himalayas\", \"price\": \"£45.17\"}\n\n---\n\n## Item 2\n\n{\"title\": \"Full Moon over Noah's Ark\", \"price\": \"£49.43\"}\n\n---\n\n## Item 3\n\n...", "ai_extraction_failed": false } } ``` *** ### Pattern 2: Click to a category using plain English, then forEach with pagination When you do not know the CSS selector for a navigation element, describe it in plain English using the `value` field on a `click` action. This is the same `click` action covered in the Actions Reference — no special setup needed. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com", "actions": [ { "type": "click", "value": "Science category in the left sidebar" }, { "type": "forEach", "observe": "Find all product cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 12, "itemPrompt": "Extract book title and price. Return as JSON: {title, price}", "pagination": { "nextSelector": "li.next > a", "maxPages": 1 } } ] }] } ``` *** ### Pattern 3: Scrape multiple categories at the same time Pass up to 3 URL objects in a single request and they are all processed in parallel. Each URL has its own forEach config. ```json theme={null} { "urls": [ { "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 4, "itemPrompt": "Return JSON: {title, price, category: 'Mystery'}" }] }, { "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 4, "itemPrompt": "Return JSON: {title, price, category: 'Travel'}" }] }, { "url": "https://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 4, "itemPrompt": "Return JSON: {title, price, category: 'Historical Fiction'}" }] } ] } ``` The response `data` array has three entries, one per URL, each with their 4 extracted books. Maximum 3 URLs per request. Each URL counts as 1 credit. *** ### Pattern 4: Click to category, navigate each book, scroll to load full description Pre-action clicks into a category. forEach navigates into each book's detail page. A per-element scroll reveals the full description. The AI extracts everything. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com", "actions": [ { "type": "click", "selector": "a[href='catalogue/category/books/poetry_23/index.html']" }, { "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 3, "waitAfterClick": 1000, "actions": [ { "type": "scroll", "to": "50%" } ], "itemPrompt": "Extract the book title, price, star rating (One through Five), and the full product description paragraph. Return as JSON: {title, price, star_rating, description}" } ] }] } ``` What happens step by step: 1. Opens the homepage in a real browser 2. Clicks the Poetry category link 3. Finds all book title links on the category page 4. For each book (up to 3): opens the book page, waits 1 second, scrolls down 50% to reveal the full description, captures the product section, runs AI to extract the four fields 5. Combines all results into a single output with numbered items *** ## Response Format All forEach results are returned in `result.content`. Items are numbered from 1 and separated by `---`. Without `itemPrompt`, you get the raw captured content per item: ``` ## Item 1 # Sharp Objects Price: £47.82 Rating: Four stars Stock: In stock A gripping psychological thriller... --- ## Item 2 # In a Dark, Dark Wood Price: £19.63 ... ``` With `itemPrompt`, each item has already been extracted by the AI before being combined: ``` ## Item 1 {"title": "Sharp Objects", "price": "£47.82", "star_rating": "Four", "availability": "In stock"} --- ## Item 2 {"title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One", "availability": "In stock"} ``` If you also provide a top-level `prompt` with `output: "json"`, the AI reads the combined output and returns a clean final JSON array in the `content` field of the response. *** ## itemPrompt vs Top-level prompt Both are optional but they serve different purposes and work well together. | | `itemPrompt` | Top-level `prompt` | | -------------------- | -------------------------------------------------- | ---------------------------------------------------------- | | When it runs | Right after each item is captured, during scraping | Once, after all items are collected and combined | | What it sees | Only that one item's content | All items together | | Best for | Per-item field extraction, cleaning up noisy pages | Final restructuring, filtering, or summarising all results | | Where output appears | Each item's section in `result.content` | `result.content` in the response | They do not conflict. They run at completely different stages. `itemPrompt` runs during scraping, one item at a time. The top-level `prompt` runs after all scraping is finished, on the full combined output. If you use both, `itemPrompt` cleans up each item first, then the top-level prompt does a final pass on all the cleaned results together. *** ## Limits | Setting | Default | Maximum | | --------------------- | ------- | ------- | | `maxItems` | 50 | 50 | | `pagination.maxPages` | 5 | 10 | | URLs per request | 1 | 3 | *** ## Full forEach Field Reference ```json theme={null} { "actions": [{ "type": "forEach", "observe": "Describe the elements to find, e.g. Find all product cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 20, "waitAfterClick": 1000, "actions": [ { "type": "scroll", "to": "50%" } ], "itemPrompt": "Extract X, Y, Z. Return as JSON: {x, y, z}", "pagination": { "nextSelector": "li.next > a", "maxPages": 3 } }] } ``` | Field | Type | Required | Description | | ------------------------- | ------ | ----------------------- | ------------------------------------------------------------------------------------------------ | | `observe` | string | Yes | Plain English description of the elements to find and process | | `mode` | string | No | How to interact with each element. One of `inline`, `navigate`, or `click`. Defaults to `click`. | | `captureSelector` | string | No | CSS selector or plain English description of which part of the page to capture per item | | `maxItems` | number | No | Maximum number of elements to process. Default is 50, maximum is 50. | | `waitAfterClick` | number | No | Milliseconds to wait after clicking or navigating before capturing. Default is 2500. | | `actions` | array | No | Browser actions to run on each item after clicking or navigating, before capturing | | `itemPrompt` | string | No | AI extraction prompt to run on each item individually | | `pagination.nextSelector` | string | Yes if using pagination | CSS selector or plain English description of the next page button | | `pagination.maxPages` | number | No | How many additional pages to process beyond the first. Default is 5, maximum is 10. | # Browser Actions Examples Source: https://docs.spidra.io/features/actions-examples Copy-paste examples for every browser action pattern. Run against real sites instantly. All examples on this page work against [books.toscrape.com](https://books.toscrape.com) and [quotes.toscrape.com](https://quotes.toscrape.com), two public sites built for scraping practice. Copy any example, paste it into the API, and it will work as shown. *** ## 1. Simple Page Scraping: No Actions Needed When the content you want is fully visible on one page, you do not need actions at all. Just send the URL with a prompt and let the AI extract what you need. **Use this when:** the list is short, fits on one page, and does not require any clicking. ```json theme={null} { "urls": [{ "url": "https://quotes.toscrape.com" }], "prompt": "List every quote and its author. Return as a JSON array: [{quote, author}]", "output": "json" } ``` **Response:** ```json theme={null} { "content": [ { "quote": "The world as we have created it is a process of our thinking...", "author": "Albert Einstein" }, { "quote": "It is our choices, Harry, that show what we truly are...", "author": "J.K. Rowling" }, ... ] } ``` The page has 10 quotes. A prompt handles all 10 easily without any forEach. If the same site had 200 quotes across 20 pages, that is when you would reach for forEach with pagination. *** ## 2. Dismiss a banner, then scrape Some pages load a cookie banner or modal that blocks the content. Click it away before the scrape runs. **Use this when:** a consent banner or popup is covering the content. ```json theme={null} { "urls": [{ "url": "https://example.com/products", "actions": [ { "type": "click", "value": "Accept all cookies button" }, { "type": "wait", "duration": 800 } ] }], "prompt": "List all product names and prices" } ``` *** ## 3. Search, wait for results, then scrape Type a search query and scrape the results page. **Use this when:** the content only appears after submitting a search form. ```json theme={null} { "urls": [{ "url": "https://example.com", "actions": [ { "type": "type", "selector": "input[name='q']", "value": "wireless headphones" }, { "type": "click", "selector": "button[type='submit']" }, { "type": "wait", "duration": 2000 } ] }], "prompt": "Extract every product name, price, and rating from the search results" } ``` *** ## 4. Inline forEach: Collect a List with itemPrompt Process every card on a page without navigating away from it. Each card's content is read directly and passed through `itemPrompt` for AI extraction. **Use this when:** all the data is visible on the listing page itself and you want a clean, structured result per item. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards in the product grid", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 10, "itemPrompt": "Extract the book title, price, and star rating (One/Two/Three/Four/Five). Return as JSON: {title, price, star_rating}" }] }] } ``` **Response (`result.content`):** ``` ## Item 1 {"title": "Sharp Objects", "price": "£47.82", "star_rating": "Four"} --- ## Item 2 {"title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One"} --- ## Item 3 {"title": "When We Collided", "price": "£31.77", "star_rating": "One"} ``` *** ## 5. Inline forEach with Pagination: Collect Across Multiple Pages When a listing spans multiple pages, use `pagination` to keep collecting after the first page is exhausted. **Use this when:** the catalogue has a "Next" button and you need more items than fit on one page. ```json theme={null} { "urls": [{ "url": "https://quotes.toscrape.com", "actions": [{ "type": "forEach", "observe": "Find all quote blocks on the page", "mode": "inline", "captureSelector": ".quote", "maxItems": 30, "itemPrompt": "Extract the quote text and author name. Return as JSON: {quote, author}", "pagination": { "nextSelector": "li.next > a", "maxPages": 3 } }] }] } ``` This collects up to 30 quotes, following the Next link across up to 3 extra pages. It stops as soon as it hits 30 total or runs out of pages. **Response (`result.content`):** ``` ## Item 1 {"quote": "The world as we have created it is a process of our thinking...", "author": "Albert Einstein"} --- ## Item 2 {"quote": "It is our choices, Harry, that show what we truly are...", "author": "J.K. Rowling"} --- ... ## Item 30 {"quote": "...", "author": "..."} ``` *** ## 6. Navigate Mode: Follow Links to Detail Pages Click into each product page to capture the rich detail that only exists there. This gives you descriptions, specifications, availability, and anything else that is not on the listing page. **Use this when:** the listing page only shows a preview and the full content is on the individual item page. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 6, "waitAfterClick": 800, "itemPrompt": "Extract the book title, price, star rating (One through Five), and whether it is in stock. Return as JSON: {title, price, star_rating, availability}" }] }] } ``` **Response (`result.content`):** ``` ## Item 1 { "title": "Sharp Objects", "price": "£47.82", "star_rating": "Four", "availability": "In stock" } --- ## Item 2 { "title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One", "availability": "In stock" } ``` Navigate mode loads a full page per item, so it is slower than inline. Keep `maxItems` reasonable (6–10) unless you have a lot of time to spare. *** ## 7. Click a category first, then forEach navigate Use an action to navigate to a specific section of the site before the forEach starts. The forEach runs on whatever page the browser is on after your pre-actions finish. **Use this when:** the items you want are behind a category link, tab, or filter on the homepage. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com", "actions": [ { "type": "click", "selector": "a[href='catalogue/category/books/travel_2/index.html']" }, { "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 4, "waitAfterClick": 800, "itemPrompt": "Extract the book title and price. Return as JSON: {title, price}" } ] }] } ``` **Response (`result.content`):** ``` ## Item 1 {"title": "It's Only the Himalayas", "price": "£45.17"} --- ## Item 2 {"title": "Full Moon over Noah's Ark", "price": "£49.43"} ``` *** ## 8. Click using plain English, then forEach with pagination When you do not know the exact CSS selector for a navigation element, describe it in plain English using the `value` field on a `click` action. Spidra uses AI to locate the element and click it, then the forEach runs on the resulting page. **Use this when:** the element you need to click does not have a stable CSS selector or is easier to describe in words. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com", "actions": [ { "type": "click", "value": "Science category in the left sidebar" }, { "type": "forEach", "observe": "Find all product cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 20, "itemPrompt": "Extract book title and price. Return as JSON: {title, price}", "pagination": { "nextSelector": "li.next > a", "maxPages": 1 } } ] }] } ``` *** ## 9. Scrape multiple categories in one request Pass up to 3 URLs in a single request. Spidra processes all of them in parallel and returns one result per URL. **Use this when:** you need data from several categories, brands, or pages at the same time. ```json theme={null} { "urls": [ { "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 4, "itemPrompt": "Return JSON: {title, price, category: 'Mystery'}" }] }, { "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 4, "itemPrompt": "Return JSON: {title, price, category: 'Travel'}" }] }, { "url": "https://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html", "actions": [{ "type": "forEach", "observe": "Find all book cards", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 4, "itemPrompt": "Return JSON: {title, price, category: 'Historical Fiction'}" }] } ] } ``` The response `data` array has three entries, one per URL, each with their own list of books. The whole thing runs in parallel, not sequentially. *** ## 10. Navigate + per-item scroll to reveal hidden content After navigating to each item's page, scroll down before capturing. Some pages lazy-load their content or hide a full description below the fold. **Use this when:** the destination page has content that only appears after scrolling, such as full product descriptions, reviews, and spec tables. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com", "actions": [ { "type": "click", "selector": "a[href='catalogue/category/books/poetry_23/index.html']" }, { "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 3, "waitAfterClick": 1000, "actions": [ { "type": "scroll", "to": "50%" } ], "itemPrompt": "Extract the book title, price, star rating (One through Five), and the full product description paragraph. Return as JSON: {title, price, star_rating, description}" } ] }] } ``` **What happens step by step:** 1. Opens the homepage 2. Clicks the Poetry category link 3. Finds all book title links 4. For each book (up to 3): navigates to the book page, waits 1 second, scrolls to 50% to load the full description, captures the product area, runs AI extraction 5. Combines all results into numbered items **Response (`result.content`):** ``` ## Item 1 { "title": "Poetry Unbound: 50 Poems to Open Your World", "price": "£23.00", "star_rating": "Five", "description": "Selected and introduced by Padraig O Tuama, this anthology brings together 50 poems that crack open the world..." } ``` *** ## 11. Click Mode: Expand Items and Capture Modal Content For pages where clicking an item opens a modal, drawer, or expanded section (such as hotel room cards or FAQ accordions), use `click` mode to open each one, capture its content, and move on. **Use this when:** the detail content only appears after clicking an element on the page, inside the same page. ```json theme={null} { "urls": [{ "url": "https://hotels.example.com/hotel/grand-plaza", "actions": [{ "type": "forEach", "observe": "Find all room type cards", "mode": "click", "captureSelector": "[role='dialog']", "maxItems": 8, "waitAfterClick": 1200, "itemPrompt": "Extract the room name, bed type, price per night, and list of amenities. Return as JSON: {room, bed_type, price_per_night, amenities}" }] }] } ``` **Response (`result.content`):** ``` ## Item 1 { "room": "Deluxe King Room", "bed_type": "1 King Bed", "price_per_night": "$189", "amenities": ["Free WiFi", "City view", "Air conditioning", "Mini bar"] } --- ## Item 2 { "room": "Standard Twin Room", "bed_type": "2 Twin Beds", "price_per_night": "$129", "amenities": ["Free WiFi", "Garden view", "Air conditioning"] } ``` If no modal appears after clicking, Spidra falls back to capturing the full page. *** ## 12. Full Pipeline: Navigate, Paginate, and Extract Combine forEach pagination with a top-level prompt to get a final clean output. `itemPrompt` handles per-item extraction during scraping. The top-level `prompt` does a final pass to restructure all items together. **Use this when:** you want structured data from a multi-page catalogue in one clean API response. ```json theme={null} { "urls": [{ "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "actions": [{ "type": "forEach", "observe": "Find all book title links in the product grid", "mode": "navigate", "captureSelector": "article.product_page", "maxItems": 20, "waitAfterClick": 800, "itemPrompt": "Extract title, price, star rating (One through Five), availability. Return as JSON: {title, price, star_rating, availability}", "pagination": { "nextSelector": "li.next > a", "maxPages": 2 } }] }], "prompt": "Return a clean JSON array of all books. Sort by price ascending.", "output": "json" } ``` `itemPrompt` runs on each book page as it is scraped. When all 20 are collected, the top-level `prompt` takes the combined output and sorts the final list. **Response:** ```json theme={null} { "content": [ { "title": "...", "price": "£13.99", "star_rating": "Three", "availability": "In stock" }, { "title": "...", "price": "£15.00", "star_rating": "Five", "availability": "In stock" }, ... ] } ``` *** ## Choosing the right pattern | Scenario | Pattern to use | | ---------------------------------------------- | ------------------------------------------------------------------------- | | Short list, all on one page | Top-level `prompt` only, no actions needed | | Content behind a cookie banner or login | `click` pre-action to dismiss, then scrape | | Long list across multiple pages | `inline` forEach with `pagination` | | Detail data only on individual item pages | `navigate` mode forEach | | Content hidden behind a click/expand | `click` mode forEach | | Category navigation before scraping | `click` pre-action (CSS selector or plain English), then forEach | | Multiple categories at once | Up to 3 URLs in one request, each with forEach | | Lazy-loaded or below-the-fold content | Per-element `scroll` action inside forEach | | Large dataset with structured field extraction | `itemPrompt` on each item + optional top-level `prompt` for final shaping | # API Keys Source: https://docs.spidra.io/features/api-keys Create, name, and revoke Spidra API keys from the dashboard. Pass your key as a Bearer token on API requests. Every request you make to the Spidra API must include an API key. You manage your keys from the Settings page in your dashboard under the **API Keys** tab. Spidra API Keys settings page ## Creating an API Key 1. Go to [Settings](https://app.spidra.io/dashboard/settings?tab=api) in your dashboard. Spidra dashboard settings navigation 2. Click **Add New API Key**. 3. Give the key a name so you can identify it later, for example "Production" or "Local Dev". 4. Click **Create API Key**. The key appears in your list immediately. You can copy it using the copy icon next to the masked value. ## Using Your API Key Pass your API key as a Bearer token: ```bash theme={null} curl -X POST https://api.spidra.io/api/scrape \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": [{"url": "https://example.com"}], "prompt": "Extract the page title"}' ``` See the [API Reference](/api-reference/introduction) for full details on authentication. ## Managing Your Keys You can have up to **5 API keys** on your account. The Settings page shows a usage bar so you can see how many you have active. ### Disabling a key Toggle the **Active** switch next to any key to disable it without deleting it. Disabled keys are rejected by the API. This is useful if you suspect a key has been exposed but want to keep it around temporarily. Toggle API key active state ### Deleting a key Click the delete icon next to a key to remove it permanently. Any running integration or application using that key will stop working immediately. You cannot delete your last remaining key. You must always have at least one key on your account. # Authenticated Scraping Source: https://docs.spidra.io/features/authenticated-scraping Scrape content behind login walls using session cookies Some websites require you to be logged in to see certain content. Authenticated scraping lets you access this content by providing your session cookies to Spidra. *** ## When You Need It Use authenticated scraping when: * The content is behind a login page * You see different data when logged in vs logged out * The website requires a subscription or account * You want to scrape your own account data *** ## How It Works 1. Log into the target website in your browser 2. Copy your session cookies 3. Include them in your API request 4. Spidra uses those cookies to access the page as if it were you *** ## Getting Your Cookies ### Step 1: Log In Open the website in Chrome, Firefox, or Edge and log into your account. ### Step 2: Open DevTools Press `F12` or right-click → Inspect → go to the **Application** tab (Chrome) or **Storage** tab (Firefox). ### Step 3: Find Cookies Click **Cookies** in the left sidebar, then select the website domain. ### Step 4: Copy Cookies You have two options: **Option A - Copy specific cookies:** Look for authentication cookies (often named `session`, `auth`, `token`, or similar). Copy the name and value of each. **Option B - Copy all cookies:** Select all rows (Ctrl/Cmd+A), then copy (Ctrl/Cmd+C). *** ## Cookie Formats Spidra accepts cookies in two formats. The API auto-detects which format you're using. ### Standard Format The traditional `name=value` format, separated by semicolons: ```json theme={null} { "urls": [{"url": "https://app.example.com/dashboard"}], "prompt": "Extract my account details", "output": "json", "cookies": "session_id=abc123xyz; auth_token=eyJhbGciOiJ..." } ``` ### Raw DevTools Format Paste directly from Chrome DevTools without any formatting: ```json theme={null} { "urls": [{"url": "https://app.example.com/dashboard"}], "prompt": "Extract my account details", "output": "json", "cookies": "session_id\tabc123xyz\t.example.com\t/\t2026-12-31\t50\t✓\t✓\tLax\nauth_token\teyJhbGciOiJ...\t.example.com\t/\t2026-12-31\t200\t✓\t✓\tStrict" } ``` The API extracts the cookie name, value, domain, and path from each row. Extra columns like Size, HttpOnly, and SameSite are ignored. *** ## Examples ### Scraping a dashboard ```json theme={null} { "urls": [{"url": "https://analytics.example.com/reports"}], "prompt": "Extract the monthly traffic summary", "output": "json", "cookies": "session=eyJ...; csrf_token=abc123" } ``` ### Crawling authenticated pages ```json theme={null} { "baseUrl": "https://app.example.com/projects", "crawlInstruction": "Find all project pages", "transformInstruction": "Extract project name, status, and last updated date", "maxPages": 20, "cookies": "auth=eyJ...; user_id=12345" } ``` *** ## Tips * **Session expiry**: Cookies expire. If your scrape fails with authentication errors, get fresh cookies. * **Required cookies**: You usually don't need all cookies. Look for ones with names like `session`, `auth`, `token`, `jwt`, or `user`. * **Domain matching**: Cookies are domain-specific. Make sure you're copying cookies from the correct domain. * **Incognito test**: Before scraping, try opening the URL in an incognito window with your cookies to verify they work. *** ## Legal Responsibility You are responsible for ensuring your authenticated scraping complies with applicable laws and the website's Terms of Service. Only scrape content you're authorized to access. Cookies are processed transiently and never stored by Spidra. *** API reference Crawl with authentication # Batch Scrape Source: https://docs.spidra.io/features/batch-scraping Scrape up to 50 URLs in a single request with full per-item results, credit tracking, and retry support Batch scraping lets you queue up to **50 URLs** in one API call. Each URL is processed independently and in parallel. You get back per-item results with status, content, credits used, and timestamps — all under a single `batchId`. Use batch scraping when: * You have a list of product, article, or listing URLs to extract * You want one API call per dataset rather than managing dozens of individual jobs * You need to retry only the URLs that failed without re-running the whole set *** ## How It Works Send a `POST /api/batch/scrape` with your URL list and extraction options. You get a `batchId` back immediately — the job is queued. Spidra processes each URL independently using a real browser. CAPTCHA solving, proxy routing, and AI extraction all run per-item. Call `GET /api/batch/scrape/{batchId}` every few seconds. The response includes live progress counters (`completedCount`, `failedCount`) and per-item results. If any items fail, call `POST /api/batch/scrape/{batchId}/retry`. Only the failed items are re-queued — successful ones are untouched. *** ## Quick Start ```bash cURL theme={null} # 1. Submit curl -X POST https://api.spidra.io/api/batch/scrape \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com/product/1", "https://example.com/product/2", "https://example.com/product/3" ], "prompt": "Extract the product name, price, and availability", "output": "json" }' # Response: # { "status": "queued", "batchId": "abc-123", "total": 3 } # 2. Poll until done curl https://api.spidra.io/api/batch/scrape/abc-123 \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```javascript Node.js theme={null} async function scrapeProducts(urls) { // Submit const submit = await fetch("https://api.spidra.io/api/batch/scrape", { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json", }, body: JSON.stringify({ urls, prompt: "Extract the product name, price, and availability", output: "json", }), }); const { batchId } = await submit.json(); // Poll while (true) { const status = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}`, { headers: { Authorization: "Bearer YOUR_API_KEY" } } ); const data = await status.json(); if (["completed", "failed", "cancelled"].includes(data.status)) { return data.items; } console.log(`${data.completedCount}/${data.totalUrls} done...`); await new Promise((r) => setTimeout(r, 3000)); } } ``` ```python Python theme={null} import time import requests BASE = "https://api.spidra.io/api" HEADERS = {"Authorization": "Bearer YOUR_API_KEY"} def scrape_products(urls): # Submit resp = requests.post( f"{BASE}/batch/scrape", headers=HEADERS, json={ "urls": urls, "prompt": "Extract the product name, price, and availability", "output": "json", }, ) batch_id = resp.json()["batchId"] # Poll while True: status = requests.get( f"{BASE}/batch/scrape/{batch_id}", headers=HEADERS ).json() if status["status"] in ("completed", "failed", "cancelled"): return status["items"] print(f"{status['completedCount']}/{status['totalUrls']} done...") time.sleep(3) ``` *** ## Polling Pattern Batch jobs are asynchronous. Poll `GET /api/batch/scrape/{batchId}` every **2–5 seconds** until `status` is a terminal value. | `status` | Meaning | | ----------- | --------------------------------------------------------------- | | `pending` | Queued, no items have started yet | | `running` | At least one item is being processed | | `completed` | All items finished (some may have failed — check `failedCount`) | | `failed` | The entire batch failed unexpectedly | | `cancelled` | You cancelled it via `DELETE /api/batch/scrape/{batchId}` | `completed` does not mean every URL succeeded. A batch is `completed` when all items have reached a terminal state (`completed` or `failed`). Always check `failedCount` and inspect individual item statuses. *** ## Per-Item Results Each item in the `items` array represents one URL: ```json theme={null} { "uuid": "item-uuid", "url": "https://example.com/product/1", "jobId": "bull-job-id", "status": "completed", "result": { "name": "Widget Pro", "price": 49.99, "available": true }, "error": null, "creditsUsed": 3, "startedAt": "2024-01-15T10:00:01Z", "finishedAt": "2024-01-15T10:00:06Z", "screenshotUrl": null } ``` | Field | Description | | --------------- | ------------------------------------------------------------------------------ | | `uuid` | Unique ID for this batch item | | `url` | The URL that was scraped | | `status` | `pending`, `running`, `completed`, or `failed` | | `result` | Extracted content (object if JSON, string if markdown). `null` until completed | | `error` | Error message if `status` is `failed`, otherwise `null` | | `creditsUsed` | Credits consumed by this item. `0` for failed items | | `startedAt` | When the worker picked up this item | | `finishedAt` | When this item reached a terminal state | | `screenshotUrl` | S3 URL if `screenshot: true` was set, otherwise `null` | *** ## Structured Output Pass a `schema` to enforce a specific output shape across all URLs in the batch. The AI will return JSON matching your schema for every item. ```json theme={null} { "urls": [ "https://shop.example.com/item/1", "https://shop.example.com/item/2" ], "prompt": "Extract the product details", "schema": { "type": "object", "required": ["name", "price"], "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "currency": { "type": ["string", "null"] }, "available": { "type": ["boolean", "null"] } } } } ``` When a `schema` is provided, `output` is automatically set to `"json"`. The schema is validated before the batch is queued — a `422` is returned if it is malformed. Full guide on nested objects, arrays, nullable fields, and schema limits *** ## Retrying Failed Items When a batch completes with some failures, retry only those items — no need to re-run the whole batch: ```bash cURL theme={null} curl -X POST https://api.spidra.io/api/batch/scrape/abc-123/retry \ -H "Authorization: Bearer YOUR_API_KEY" # Response: # { "status": "queued", "retriedCount": 2 } ``` ```javascript Node.js theme={null} const res = await fetch( `https://api.spidra.io/api/batch/scrape/${batchId}/retry`, { method: "POST", headers: { Authorization: "Bearer YOUR_API_KEY" } } ); const { retriedCount } = await res.json(); console.log(`${retriedCount} items re-queued`); ``` ```python Python theme={null} resp = requests.post( f"{BASE}/batch/scrape/{batch_id}/retry", headers=HEADERS, ) print(f"{resp.json()['retriedCount']} items re-queued") ``` The batch status resets to `running` and you poll the same `batchId` until it completes again. Successfully completed items are never touched. *** ## Cancelling a Batch Cancel a running or pending batch to stop processing and refund credits for items that have not started yet: ```bash theme={null} curl -X DELETE https://api.spidra.io/api/batch/scrape/abc-123 \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```json theme={null} { "status": "cancelled", "cancelledItems": 8, "creditsRefunded": 16 } ``` Items already running will complete normally. Only pending items are cancelled and refunded. *** ## Proxy & Geo-Targeting Apply stealth proxy routing to every URL in the batch with `useProxy` and `proxyCountry`: ```json theme={null} { "urls": ["https://amazon.de/dp/B123", "https://amazon.de/dp/B456"], "prompt": "Extract price and availability", "output": "json", "useProxy": true, "proxyCountry": "de" } ``` Full country list, EU rotation, and billing details *** ## Cookies & Authenticated Pages Pass session cookies to scrape pages behind a login. Cookies are never stored — they are passed ephemerally to the worker and discarded after processing. ```json theme={null} { "urls": [ "https://app.example.com/reports/q1", "https://app.example.com/reports/q2" ], "cookies": "session=eyJ...; auth_token=abc123", "prompt": "Extract the report summary", "output": "json" } ``` Full guide on obtaining and formatting cookies *** Full request reference Polling and response shape See all your batch jobs Stop a batch or re-run failures # CAPTCHA Solver Source: https://docs.spidra.io/features/captcha-solver Automatic CAPTCHA and anti-bot challenge handling. Supports reCAPTCHA v2/v3, hCaptcha, Cloudflare Turnstile, DataDome, and more — runs transparently on every job. Spidra automatically detects and solves CAPTCHAs and anti-bot challenges on every scrape and crawl job. You do not need to pass any extra parameters — if a page throws a challenge, Spidra handles it in the background and continues the job. ## Supported Challenge Types | Challenge | Status | | -------------------------- | --------- | | reCAPTCHA v2 (checkbox) | Supported | | reCAPTCHA v2 (invisible) | Supported | | reCAPTCHA v3 (score-based) | Supported | | hCaptcha | Supported | | Cloudflare Turnstile | Supported | | Cloudflare JS challenge | Supported | | DataDome | Supported | | PerimeterX / HUMAN | Supported | | GeeTest | Supported | | FunCaptcha / Arkose Labs | Supported | | AWS WAF CAPTCHA | Supported | | Imperva / Incapsula | Supported | ## How It Works When Spidra's headless browser navigates to a page and encounters an anti-bot challenge, it: 1. Detects the challenge type automatically 2. Routes it to the appropriate solver 3. Submits the solution back to the page 4. Continues loading the page and runs your prompt or browser actions All of this happens inside the same job — no extra API calls, no callbacks, no manual intervention. The job takes slightly longer on pages that require solving, but the outcome is the same clean result. ## No API Parameter Required CAPTCHA solving is always on. There is nothing to enable. ```json theme={null} { "urls": [{ "url": "https://example.com/protected-page" }], "prompt": "Extract the main content", "output": "json" } ``` If the page has a CAPTCHA, it gets solved. If it does not, nothing happens and no credits are charged. ## Seeing Whether a CAPTCHA Was Solved The job status response includes a `captchaSolvedCount` field in `result.stats`: ```json theme={null} { "status": "completed", "result": { "content": { ... }, "stats": { "durationMs": 12400, "captchaSolvedCount": 1, "inputTokens": 2800, "outputTokens": 140, "totalTokens": 2940 } } } ``` `captchaSolvedCount: 1` means one challenge was encountered and solved during the job. If the value is `0`, the page either had no challenge or the challenge was bypassed without a solve (common with Cloudflare JS challenges that pass after browser fingerprint checks). ## Billing Each solved CAPTCHA costs **10 credits**. This is charged on top of the base 2 credits per URL. * Page with no CAPTCHA: 2 credits * Page with 1 CAPTCHA: 12 credits (2 base + 10 solve) * Page with 2 CAPTCHAs (e.g., one on load, one on form submit): 22 credits Credits are only charged for successful solves. If a challenge appears but cannot be solved, no CAPTCHA credit is deducted — though the job may fail depending on the site. You can monitor CAPTCHA usage and credit consumption from the **Usage** page in the dashboard. ## Combining with Stealth Mode For sites with aggressive bot detection — Cloudflare, DataDome, Akamai, PerimeterX — combine CAPTCHA solving with Stealth Mode. Stealth Mode routes the request through a residential proxy, making it look like a real user. CAPTCHA solving handles any challenges that still appear. ```json theme={null} { "urls": [{ "url": "https://heavily-protected-site.com/data" }], "prompt": "Extract the product listings", "output": "json", "useProxy": true, "proxyCountry": "us" } ``` Most protected sites are handled by one or the other. Some require both. Residential proxy rotation for IP-based blocks and geo-restricted content Scrape login-protected pages by injecting session cookies Click, scroll, and type to interact with pages before extraction Full API reference for the POST /scrape endpoint # Crawl Source: https://docs.spidra.io/features/crawling Crawl entire websites and extract structured data from every page. Set a starting URL, define which links to follow, and get a full dataset in one job. The Crawling Playground lets you crawl multiple pages from a website in one job. You define where to start, which pages to visit, and what to extract — Spidra handles the rest. ## How to set up a crawl ### 1. Enter a target URL This is the starting point. Spidra loads this page first and follows links from it. Good starting points are index, category, or listing pages — not individual content pages. For example, `https://example.com/blog` works better than `https://example.com/blog/my-specific-post`. Target URL field ### 2. Set crawl instructions Tell Spidra which pages to follow, in plain language. This controls which links get visited and which get ignored. Examples: * `Crawl all blog post pages` * `Visit product pages only` * `Follow links in the /docs/ section` Crawl instructions field ### 3. Set transform instructions (optional) Describe what to extract from each page. Every page in the crawl is processed with the same instruction. Examples: * `Extract the title, author, and publish date` * `Get the product name, price, and description` * `Pull the main article content` Transform instructions field If you skip this field and don't provide a schema, Spidra returns the raw markdown of each page without calling AI. No token credits are charged in this mode — the content is ready to feed into your own pipeline. ### 4. Define an output schema (optional) Use a schema when you need every page to return the same fields in the same format. Spidra's schema editor lets you define fields visually — pick a type, mark which are required, and add nested objects or arrays. When a schema is set, the AI returns JSON matching it exactly for every page. This is useful when you're storing results in a database or processing them programmatically. You can set a schema alongside a transform instruction, or use it on its own. Use the [Spidra JSON Schema Generator](https://spidra.io/tools/json-schema-generator) to build your schema visually, then paste it into the schema field. ### 5. Set max pages Cap how many pages the crawl visits. The default is 5. The maximum is 50. Max pages field Start small to validate your instructions before expanding. Once you're confident in your setup, increase the limit. ## Advanced options ### Path filters Use **Include paths** and **Exclude paths** to narrow the crawl to specific sections of a site. * **Include paths** — Only pages whose URL paths match one of your patterns are visited. For example, `/blog/*` limits the crawl to pages under `/blog/`. * **Exclude paths** — Pages matching any pattern are skipped. For example, `/tag/*` prevents tag pages from being crawled. Both support glob-style patterns. You can use both together — include paths are applied first, then excludes are applied on top. ### Max depth Limits how many links deep the crawler goes from the base URL. `0` means only the base URL itself. `1` means the base URL and pages directly linked from it. Leave it blank for unlimited depth. ### Domain options * **Allow subdomains** — Follows links to subdomains of the base domain (e.g. `docs.example.com` when your base is `example.com`). * **Crawl entire domain** — Follows any link on the same root domain, regardless of the starting path. Combine this with path filters to keep the scope manageable. * **Ignore query parameters** — Treats URLs that differ only by query string as the same page. Useful on sites that append tracking parameters to every link. ### Webhook Provide a webhook URL to receive a POST request each time a page finishes processing. This lets you stream results into your own pipeline as they come in, rather than waiting for the whole job to complete. ## Features * **Smart page discovery** — Spidra uses your crawl instruction to decide which links to follow. It ignores navigation menus, login pages, and other links that don't match what you're looking for. * **AI-powered extraction** — When a transform instruction or schema is set, each page is processed with AI to extract exactly the data you described. No selectors or XPath needed. * **Raw markdown mode** — When no transform instruction or schema is set, each page's `data` field contains the plain markdown of that page. No AI is used, no token credits are charged. Useful for feeding content into your own AI pipeline. * **Automatic CAPTCHA solving** — Spidra handles CAPTCHAs automatically during the crawl. * **Stealth mode** — Enable proxy routing to reduce blocks and rate limits on sites with bot detection. * **Download results** — After a crawl completes, download all extracted data as a ZIP file containing: the extracted content, raw markdown, and original HTML snapshots for each page. * **Retry failed pages** — If extraction fails on specific pages, retry them individually without re-crawling the whole site. ## Extract from Crawl After a crawl finishes, you can run a new extraction on the same pages without visiting the site again. Spidra reuses the HTML and markdown saved during the original crawl and runs your new prompt against it. This is useful when: * You want to pull different fields from pages you already crawled * Your first extraction prompt wasn't quite right * You need the same pages in two different formats ```json theme={null} { "transformInstruction": "Extract only the product price and availability status" } ``` Send a `POST` to `/crawl/{jobId}/extract` with a new `transformInstruction`. You'll get back a new `jobId` to poll. Full API reference for the extract endpoint ## Tips * **Be specific with crawl instructions.** `"All blog posts from 2024"` works better than `"crawl everything"`. Clear constraints lead to cleaner results and fewer wasted pages. * **Start with 3–5 pages.** Validate your crawl and transform instructions on a small set before scaling up. * **Use path filters on large sites.** `includePaths` and `excludePaths` are the fastest way to keep a crawl focused without relying entirely on the crawl instruction. * **Choose the right starting URL.** Make sure your target URL links to the pages you want. Index and listing pages are usually the right choice. # Dashboard Source: https://docs.spidra.io/features/dashboard Learn how to navigate the Spidra dashboard, visualize usage metrics, monitor recent scrapes, and access key shortcuts. The Spidra dashboard gives you a real-time overview of your scraping activity, credit usage, recent integrations, and saved presets — **all in one place**. It offers visibility into how your resources are being used. Hero Light *** ## Quick Access and Summary At the top of the dashboard, you’ll find quick stats including: * **Total Scrapes** – Number of successful scraping jobs you've run. * **Total Presets** – Count of saved configurations from the Playground. * **Total Integrations** – Number of currently active integrations with third-party platforms. Hero Light These stats give you a quick pulse on your account activity ## Usage Analytics Spidra includes a built-in credit usage chart so you can monitor your consumption over time. This chart breaks down: * **Credit Used** * **Total Requests** * **Tokens Used** (powered by LLM interactions) * **CAPTCHAs Solved** (if bypass mechanisms were triggered) Hero Light ### On the dashboard you also find the following: * **Recent Integrations**: You'll see a card showcasing your 2 most recent integrations, including their: * Integration type (e.g. Slack, Discord, Email) * Status (active/inactive) * Last run & Next run Hero Light * **Scraping Timeline**: Another card presents your Scraping Timeline, giving a high-level chronological view of recent scraping jobs. Hero Light * **Recent Presets**: The final card on the dashboard summarizes your most recently saved presets. Each card includes: preset Name & description Hero Light # Integrations Source: https://docs.spidra.io/features/integrations Set up integrations to automatically deliver scraped data to platforms like Slack, Discord, Webhook, Email, and more on a schedule Integrations is a powerful feature that makes Spidra not just a scraping platform, but an automation engine. Spidra lets you **automatically send your scraped data** to other platforms — like Slack, Discord, Telegram, or Email without writing a line of code. You can create an integration from any preset and define **when** and **where** the data should be delivered. *** ## Integration Overview On the **Integrations page**, you'll see all your existing integrations, grouped by the preset they were created from. Each integration includes: * Linked Preset * **Type** this is the integration Platform e.g., Email, Slack, Discord etc. * **Status** (active/inactive) * **Schedule** e.g. daily at 9 AM * **Quick actions**: such as **Test**, **View logs**, **Edit**, and **Delete** Hero Light ### Creating a New Integration To create a new integration: 1. Go to the Integrations Tab and click on the **Add Integration** button. Hero Light 2. **Type**: Choose a preset from the dropdown to define what you want to integrate, then select your preferred destination: Slack, Discord, Webhook, or Email. Hero Light For this guide, we will make use of [Discord](https://discord.com/) as the preferred destination. 3. **Configure the Integration**: For Discord , you’ll need to provide a webhook URL generated in your server settings. Then proceed to add the webhook URL into the input field and click on the **Next** button. Each platform has its own setup fields (e.g. webhook URL, channel name, email address). Fill in the required configuration based on your selected platform. Hero Light 4. **Schedule Integration**: Set how often you want this integration to run: * Daily * Weekly * Monthly Also choose what time it should trigger. Hero Light 5. **Create and Activate**: Once configured and scheduled, name your integration click on the **Create** button. Hero Light Your integration will appear in the integrations list, ready to deliver new scrape results based on your chosen schedule to your desired destination. We highly recommend testing your integration before activating to make sure your destination is receiving data correctly # Logs Source: https://docs.spidra.io/features/logs Monitor, debug, and download detailed records of your scraping jobs and interactions with Spidra. The **Logs** gives you full visibility into your scraping activity and LLM interactions. Whether you’re debugging a failed job, reviewing recent extractions, or exporting logs for reporting. It is important for transparency, debugging, optimization, status tracking, and ease of exploration. *** ## Logs List View The main Logs interface displays a **chronological list** of all scrape jobs and LLM requests you’ve triggered. Each row in the table includes: * **Target URL** * **Status** (`Completed`, `Pending`, `Error`) * **Duration** * **Timestamp** * **Actions** (e.g., View details, Download log) Hero Light ## Viewing Log Details Click on the **quick actions** button and select **view**, this opens a detailed sidebar showing: `URL`, `Log ID`, `Status`, `Channel`, `Tokens Used`, `Latency`, `Extraction prompt`, `Response`. Hero Light ## Download Logs Each entry includes a **download log** that exports the log in a structured format `json` file. Hero Light # Marketplace Source: https://docs.spidra.io/features/marketplace Discover and reuse published scraping presets from the Spidra community to speed up your data extraction workflows The **Spidra Marketplace** is where the community shares scraping presets you can instantly use, remix, or integrate. It is like templates for extracting data from job boards, e-commerce platforms, blogs, and more. Hero Light *** ## Explore Published Presets On the Marketplace, you'll find a searchable, filterable list of public presets for different categories such as e-commerce, news & media, social media etc. Each card includes: * Preset Title e.g *Upwork Job Scraper* * Short Description e.g " *Extract job listings including title, budget or hourly rate, experience level, and...* " * Category e.g *Job Market* * Difficulty e.g *Advanced* * Output Format e.g *JSON* * Ratings (*You can also rate the preset* ) * Example Output * Extraction Prompt * A button to try in Playground Hero Light *** ### Using a Marketplace Preset Once you find a preset you like, click to open it. A sidebar containing the preset details will appear, scroll to the bottom and click on the **Try in Playground** button. After loading a preset in the Playground, you can tweak or customize the URLs, actions, or extraction instructions to match your specific use case. Hero Light Marketplace presets are not locked. After running teh scrape you can save the modified version as your own private preset. # Scrape Source: https://docs.spidra.io/features/playground Learn how to use the Spidra Playground to run scraping tasks, configure prompts, choose output formats, and export data or SDK-ready code The **Spidra Playground** is your interactive hub for creating and running scraping jobs using natural language. It's built for flexibility with no coding knowledge required. Whether you're scraping one or multiple URLS, the Playground gives you everything you need in a single, real-time interface. *** ### Target URL(s) The first field allows you to add one or more URLs you want to scrape. Hero Light ### Operations Operations let you interact with the page before extracting content — clicking buttons, typing into fields, scrolling, or iterating over elements. Each URL has a **selector mode toggle** (CSS or AI): * **AI mode** (default) — Describe elements in plain English. For example: `Click the "Load More" button` or `Type 'laptop' into the search field`. * **CSS mode** — Use precise CSS or XPath selectors. For example: `#submit-button` or `//div[@class='productName']`. This is great for advanced users who want precise control or are debugging tricky pages. Hero Light ### Extraction Prompt This is the core LLM-powered field which describes what data you want extracted in natural language. For example: * *"Get all product titles and prices."* * *"Extract all blog post titles and their publish dates."* Hero Light The clearer your prompt, the better your results. Be specific about what fields to extract. ### Output Format Select your preferred output format from the dropdown: | Format | Description | | -------------- | ----------------------------------------- | | **Markdown** | Formatted text with headings and lists | | **Plain text** | Raw unformatted text | | **JSON** | Structured data for programmatic use | | **Table** | Tabular format for spreadsheet-style data | Hero Light ### SDK Integration Above the output panel, click on the **code** button and you’ll find a dynamically generated API code snippet. It shows you the SDK code in: `Python` `JavaScript` and `CURL`. Hero Light ### Advanced Configuration The options available include: ✅ **Stealth Mode (Proxy)** — Enables proxy rotation and anti-bot protection. When turned on, a country selector appears — pick a specific country, choose **Europe Mix** to rotate across all EU member states, or leave it on **Worldwide** for no preference. ✅ **Screenshot** — Captures a screenshot of each page after scraping. ✅ **Full Page Screenshot** — Captures the entire scrollable page instead of just the viewport. ✅ **Extract Content Only** — Removes headers, footers, and navigation from the output. ✅ **Scroll to Bottom** — Auto-scrolls page to load dynamic content. These checkboxes can be toggled per scrape. Hero Light ## Save or Scrape * **Save Preset** : Stores your current configuration (URL, actions, prompt, config) into the Presets tab. * **Start Scrape** : Runs the scrape job immediately and shows live output in the Output Section. # Presets Source: https://docs.spidra.io/features/presets Learn how to save, manage, and reuse scraping configurations in Spidra using Presets **Presets** is a powerful feature in Spidra that allows you save reusable scraping configurations such as your target URLs, operations, prompt, and output. So you don’t have to start from scratch every time. They act as **scraping templates** for personal use or for sharing with the wider Spidra community via the Marketplace. Each preset stores a full snapshot of your Playground configuration: | Configuration | Description | | --------------------- | ---------------------------------------------- | | **Target URLs** | One or more URLs to scrape | | **Operations** | Any browser-like actions (click, scroll, type) | | **Prompt** | Your LLM prompt describing what to extract | | **Advanced Settings** | Stealth mode, scroll to bottom, etc. | | **Output Format** | JSON or Markdown | | **Extracted Result** | The actual response output at time of save | ### Saving a Preset from Playground In the Playground, instead of running the scrape immediately, you can hit: **Save Preset**. This captures everything from your current session and stores it. Hero Light ### Viewing Your Presets Presets appear in the **Presets tab** as a list of cards. Each card contains: * Preset Name * Short Description * Tag or icon (if published) * Quick actions (edit, integration, delete) * A **load in playground** button for you to rerun the preset the playground. Hero Light # Stealth Mode Source: https://docs.spidra.io/features/stealth-mode Bypass bot detection with residential proxy rotation across 50+ countries. Handles Cloudflare, Akamai, and IP-based blocks automatically. Stealth Mode routes your scrape and crawl requests through a network of residential proxies, making them appear to come from real users in real locations. Use it when a site blocks direct requests, requires a specific country IP, or shows different content based on location. ## When to Use It * The target site returns CAPTCHA pages or blocks automated requests * You need to see prices, content, or search results from a specific country * The site detects and blocks datacenter IPs * You're scraping a site that uses bot-protection services like Cloudflare or Akamai ## How to Enable It ### In the Playground Toggle **Stealth Mode** in the Advanced Configuration section. Once enabled, a country selector appears — pick a specific country, choose **Europe Mix** to rotate across EU member states, or leave it on **Worldwide** for no preference. ### Via API Add `"useProxy": true` to any scrape or crawl request. Optionally add `"proxyCountry"` to target a specific location. ```json theme={null} { "urls": [{"url": "https://amazon.de/dp/B123456"}], "prompt": "Extract the product price in euros", "output": "json", "useProxy": true, "proxyCountry": "de" } ``` ## Country Targeting Pass `"proxyCountry"` with any of the values below. Omitting `proxyCountry` (or setting it to `"global"`) lets Spidra pick the best available exit node with no location preference. Setting it to `"eu"` rotates randomly across all 27 EU member states on each request — useful when you need a European IP but don't care which specific country. For a specific country, use its two-letter ISO code: | Code | Country | Code | Country | Code | Country | | ---- | -------------- | ---- | -------------- | ---- | ------------ | | `us` | United States | `de` | Germany | `jp` | Japan | | `gb` | United Kingdom | `fr` | France | `au` | Australia | | `ca` | Canada | `br` | Brazil | `in` | India | | `nl` | Netherlands | `sg` | Singapore | `es` | Spain | | `it` | Italy | `mx` | Mexico | `za` | South Africa | | `ng` | Nigeria | `ar` | Argentina | `be` | Belgium | | `ch` | Switzerland | `cl` | Chile | `cn` | China | | `co` | Colombia | `cz` | Czech Republic | `dk` | Denmark | | `eg` | Egypt | `fi` | Finland | `gr` | Greece | | `hk` | Hong Kong | `hu` | Hungary | `id` | Indonesia | | `ie` | Ireland | `il` | Israel | `kr` | South Korea | | `my` | Malaysia | `no` | Norway | `nz` | New Zealand | | `pe` | Peru | `ph` | Philippines | `pl` | Poland | | `pt` | Portugal | `ro` | Romania | `sa` | Saudi Arabia | | `se` | Sweden | `th` | Thailand | `tr` | Turkey | | `tw` | Taiwan | `ua` | Ukraine | `vn` | Vietnam | ## Examples ### Scraping geo-restricted content ```json theme={null} { "urls": [{"url": "https://example.com/pricing"}], "prompt": "Extract the pricing tiers and amounts", "output": "json", "useProxy": true, "proxyCountry": "us" } ``` ### Crawling with a European IP ```json theme={null} { "baseUrl": "https://example.eu/products", "crawlInstruction": "Find all product pages", "transformInstruction": "Extract product name and price in EUR", "maxPages": 10, "useProxy": true, "proxyCountry": "eu" } ``` ### No country preference ```json theme={null} { "urls": [{"url": "https://example.com"}], "prompt": "Extract the main content", "output": "json", "useProxy": true } ``` ## Billing Stealth Mode is billed against your **bandwidth quota**, not your credit balance. Each request routed through the proxy network consumes bandwidth (measured in MB) based on the size of the pages fetched. There is no extra per-URL credit charge for enabling `useProxy`. Monthly bandwidth included per plan: | Plan | Included bandwidth | | ------- | ------------------ | | Free | 50 MB | | Starter | 500 MB | | Builder | 2 GB | | Pro | 5 GB | If you run out of bandwidth mid-cycle, proxy requests are paused until your quota renews or you purchase a top-up. You can monitor your bandwidth usage from the **Usage** page in the dashboard. Scrape API reference Crawl API reference Buy more bandwidth # Structured Output Source: https://docs.spidra.io/features/structured-output Guarantee exact JSON shapes from every scrape. Define a schema and get consistent field names, types, and null handling — works with Zod and Pydantic. ## What is structured data? When you scrape with a `prompt`, the AI reads the page and returns whatever JSON it decides makes sense. The shape can vary between runs. A field might appear with a different name. A field might be missing if the AI was not confident. If you are saving results to a database or processing them in code, this inconsistency is a problem. Structured output solves this. You add a `schema` to your request that describes the exact shape you want. The AI must return JSON that matches that shape exactly, with the field names, types, and nesting you defined. If the AI cannot find a value for a field, it writes `null` instead of skipping the field. **Without a schema**, asking for job details might give you: ```json theme={null} { "Job Title": "Engineer", "pay": "$140k", "remote_ok": "yes" } ``` **With a schema**, you get exactly what you defined: ```json theme={null} { "title": "Engineer", "salary": 140000, "remote": true } ``` Same data, but the shape is now guaranteed. This is also available via the Spidra dashboard interface: JSON Schema Example Build and preview your schema visually — no JSON knowledge required. Copy the output and paste it directly into your request. ## Basic example Add a `schema` field to your scrape request alongside your `prompt`. The schema is a standard JSON Schema object. ```json theme={null} { "urls": [{ "url": "https://jobs.example.com/senior-engineer" }], "prompt": "Extract the job details from this posting", "schema": { "type": "object", "required": ["title", "company", "remote", "employment_type"], "properties": { "title": { "type": "string" }, "company": { "type": "string" }, "location": { "type": ["string", "null"] }, "remote": { "type": ["boolean", "null"] }, "salary_min": { "type": ["number", "null"] }, "salary_max": { "type": ["number", "null"] }, "employment_type": { "type": ["string", "null"], "enum": ["full_time", "part_time", "contract", null] }, "skills": { "type": "array", "items": { "type": "string" } } } } } ``` The response `result.content` will look like this when all the data is on the page: ```json theme={null} { "title": "Senior Software Engineer", "company": "Acme Corp", "location": "Austin, TX", "remote": true, "salary_min": 140000, "salary_max": 180000, "employment_type": "full_time", "skills": ["Python", "React", "PostgreSQL", "Docker"] } ``` And like this when salary information is not mentioned on the page: ```json theme={null} { "title": "Senior Software Engineer", "company": "Acme Corp", "location": "Austin, TX", "remote": true, "salary_min": null, "salary_max": null, "employment_type": "full_time", "skills": ["Python", "React"] } ``` The shape is the same either way. `null` means the AI looked for it and could not find it. The field is always there. Skip writing JSON Schema by hand. Generate it directly from your existing Zod or Pydantic model. ## The required rule This is the most important thing to understand about how structured data works. **Fields listed in `required` are always in the output.** If the AI cannot find a value, it writes `null`. The field is never missing. **Fields not in `required` may be omitted.** If the AI has no evidence for an optional field, it leaves it out of the response entirely rather than guessing. A concrete example. Say your schema has these properties: ```json theme={null} "properties": { "title": { "type": "string" }, "company": { "type": "string" }, "salary": { "type": ["number", "null"] }, "benefits": { "type": ["string", "null"] } } ``` If `required` is `["title", "company"]`, and the page has no salary or benefits info: ```json theme={null} { "title": "Engineer", "company": "Acme" } ``` The optional fields are completely absent. If `required` is `["title", "company", "salary", "benefits"]`, and the page still has no salary or benefits info: ```json theme={null} { "title": "Engineer", "company": "Acme", "salary": null, "benefits": null } ``` The fields are there, just null. **Rule of thumb:** put a field in `required` when you need it to always be present in your output, even as null. Leave it out of `required` when you are fine with it being absent if there is nothing to extract. ## Nullable fields To make a field nullable, pass the type as an array that includes `"null"`: ```json theme={null} { "type": ["string", "null"] } ``` This means the field can be a string or null. If the AI finds the value, it writes the string. If not, it writes `null`. This works for all types: ```json theme={null} { "type": ["number", "null"] } { "type": ["boolean", "null"] } { "type": ["string", "null"] } ``` ## Enum fields Use `enum` to restrict a field to a specific set of values. Include `null` in the enum list to allow null as a valid value. ```json theme={null} { "type": ["string", "null"], "enum": ["full_time", "part_time", "contract", null] } ``` The AI will pick the closest matching option from your list. If nothing fits, it uses `null`. Be careful with required enum fields. If the field is in `required` and the AI cannot find clear evidence for any of your enum values, it must still write something. It will either pick the closest match or write `null` if you included it in the enum. Always include `null` in your enum when the field might not always appear on the page. ## Nested objects Your schema can include nested objects. Just define `properties` inside a `properties` value, the same way you would in any JSON Schema. ```json theme={null} { "type": "object", "required": ["title", "company"], "properties": { "title": { "type": "string" }, "company": { "type": "string" }, "salary": { "type": "object", "required": ["min", "max", "currency"], "properties": { "min": { "type": ["number", "null"] }, "max": { "type": ["number", "null"] }, "currency": { "type": ["string", "null"] } } } } } ``` Output: ```json theme={null} { "title": "Senior Engineer", "company": "Acme Corp", "salary": { "min": 140000, "max": 180000, "currency": "USD" } } ``` ## Arrays of objects To extract a list of items where each item has a fixed shape, use an array with an `items` definition. ```json theme={null} { "type": "object", "required": ["products"], "properties": { "products": { "type": "array", "items": { "type": "object", "required": ["name", "price", "in_stock"], "properties": { "name": { "type": "string" }, "price": { "type": ["number", "null"] }, "in_stock": { "type": ["boolean", "null"] } } } } } } ``` Output: ```json theme={null} { "products": [ { "name": "Wireless Headphones", "price": 79.99, "in_stock": true }, { "name": "USB-C Hub", "price": 34.99, "in_stock": true }, { "name": "Laptop Stand", "price": null, "in_stock": false } ] } ``` Every item in the array has the same shape. If a product has no listed price, that item gets `null` for `price` rather than being skipped. ## Using prompt alongside schema `prompt` and `schema` are designed to work together. The schema controls the output shape. The prompt guides how the AI interprets and normalizes the page content before filling in the schema. Use `prompt` to give the AI instructions about normalization, what to look for, or what to ignore: ```json theme={null} { "urls": [{ "url": "https://jobs.example.com/engineer" }], "prompt": "Extract the job data. Normalize salary to a plain number in USD (drop symbols and commas). For employment_type, map contract-based and freelance roles to 'contract'. If the page shows salary as a range like '$140k - $180k', split into salary_min and salary_max.", "schema": { "type": "object", "required": ["title", "company", "salary_min", "salary_max", "employment_type"], "properties": { "title": { "type": "string" }, "company": { "type": "string" }, "salary_min": { "type": ["number", "null"] }, "salary_max": { "type": ["number", "null"] }, "employment_type": { "type": ["string", "null"], "enum": ["full_time", "part_time", "contract", null] } } } } ``` Think of the prompt as instructions to the AI and the schema as the contract for the output. Both are optional on their own, but they are most powerful together. ## Generating schemas with Zod or Pydantic Spidra accepts standard JSON Schema. You can write that JSON by hand, or you can use a schema validation library in your own code to generate it. **Zod (JavaScript / TypeScript)** ```typescript theme={null} import { z } from "zod"; import { zodToJsonSchema } from "zod-to-json-schema"; const JobSchema = z.object({ title: z.string(), company: z.string(), location: z.string().nullable(), remote: z.boolean().nullable(), salary_min: z.number().nullable(), salary_max: z.number().nullable(), employment_type: z.enum(["full_time", "part_time", "contract"]).nullable(), skills: z.array(z.string()), }); const jsonSchema = zodToJsonSchema(JobSchema, { $schemaUrl: false }); const response = await fetch("https://api.spidra.io/api/scrape", { method: "POST", headers: { "Content-Type": "application/json", Authorization: "Bearer your-api-key", }, body: JSON.stringify({ urls: [{ url: "https://jobs.example.com/engineer" }], prompt: "Extract the job details", schema: jsonSchema, }), }); ``` **Pydantic (Python)** ```python theme={null} from pydantic import BaseModel from typing import Optional, Literal import requests class Job(BaseModel): title: str company: str location: Optional[str] remote: Optional[bool] salary_min: Optional[float] salary_max: Optional[float] employment_type: Optional[Literal["full_time", "part_time", "contract"]] skills: list[str] json_schema = Job.model_json_schema() response = requests.post( "https://api.spidra.io/api/scrape", headers={"Authorization": "Bearer your-api-key"}, json={ "urls": [{"url": "https://jobs.example.com/engineer"}], "prompt": "Extract the job details", "schema": json_schema, }, ) ``` You define and validate your data shape in your own code using the tools you already know. The resulting JSON Schema is what you pass to Spidra. You can also use your Zod or Pydantic schema to validate the output you receive back from Spidra, giving you end-to-end type safety. ## The full response When your scrape job completes, poll `GET /scrape/{jobId}` as usual. The structured data appears in `result.content` as a parsed JSON object, not a string. ```json theme={null} { "status": "completed", "progress": { "message": "Scraping completed successfully", "progress": 1 }, "result": { "content": { "title": "Senior Software Engineer", "company": "Acme Corp", "location": "Austin, TX", "remote": true, "salary_min": 140000, "salary_max": 180000, "employment_type": "full_time", "skills": ["Python", "React", "PostgreSQL"] }, "screenshots": [], "ai_extraction_failed": false, "stats": { "durationMs": 8200, "captchaSolvedCount": 0, "inputTokens": 3100, "outputTokens": 180, "totalTokens": 3280 } } } ``` If the scrape job includes a `schema`, the job will fail rather than fall back to raw markdown when AI extraction cannot complete. This is intentional. When you pass a schema, you are expecting a specific shape, and returning unstructured markdown would be silently wrong. ## Schema warnings Some JSON Schema keywords are not supported by the AI model. If your schema includes them, Spidra strips them before processing and returns a `schema_warnings` list in the job status response so you know what was ignored. ```json theme={null} { "status": "completed", "schema_warnings": [ "Property 'title': keyword 'anyOf' is not supported and will be ignored", "Property 'salary': keyword '$ref' is not supported and will be ignored" ], "result": { ... } } ``` Warnings are non-fatal. The job still runs. But you should remove or replace the flagged keywords to make sure the AI is enforcing what you intended. **Supported keywords:** `type`, `properties`, `required`, `items`, `enum`, `nullable`, `description` **Not supported:** `$ref`, `anyOf`, `oneOf`, `allOf`, `not`, `if`, `then`, `else`, `$defs`, `definitions`, `prefixItems` ## Schema validation errors If your schema has a structural problem, the API returns a `422` error before the job is queued. No credits are used. ```json theme={null} { "status": "error", "message": "Invalid schema. Fix the errors below and try again.", "errors": [ "Root schema must be type 'object'", "Schema exceeds maximum nesting depth of 5" ] } ``` **`Root schema must be type 'object'`** Your top-level schema must be `{ "type": "object", "properties": { ... } }`. Passing an array or a plain string type at the root is not allowed. **`Schema exceeds maximum nesting depth of 5`** Your schema has more than 5 levels of nested objects. Flatten the structure or move deeply nested data into a string field that the AI formats itself. **`Schema exceeds maximum size`** The schema JSON is over 10KB. Remove unused fields or descriptions to bring it under the limit. Full API reference for the POST /scrape endpoint Combine structured output with forEach to extract lists of items # Introduction Source: https://docs.spidra.io/index Extract structured data from any website through the Playground, the API, or any of our SDKs. Spidra Documentation Spidra Documentation Spidra turns websites into structured, ready-to-use data using prompts, real browsers, proxies, CAPTCHA solving, and AI extraction without forcing users to build or maintain scraping infrastructure. ## Start here Make your first scrape in under 2 minutes through the Playground or in code. Every endpoint documented with request and response examples. ## What Spidra can do Describe what you want in plain text. Spidra opens a real browser, runs your prompt, and returns clean data. Spidra discovers pages, visits each one, and extracts structured data across the whole site automatically. Submit up to 50 URLs in one request and they all run in parallel. Define a JSON schema and get guaranteed-shape data back every time. Click, scroll, type, wait dismiss cookie banners, fill search forms, load infinite scroll content. Find every matching element on a page, follow each link to its destination, and get back a clean array. Pass session cookies to scrape pages behind login walls. Cookies are never stored. Residential proxies to 50+ countries, CAPTCHA solving, and browser fingerprint evasion built in. ## SDKs and integrations Official libraries for every major language and platform. All open source at [github.com/spidra-io](https://github.com/spidra-io). # Discord Source: https://docs.spidra.io/integrations/discord Easily connect your Spidra preset to Discord and start receiving automated scrape results in your server channels. This guide walks you through how to connect a [Spidra](https://spidra.io) preset\*\* to [Discord](https://discord.com/) in just a few simple steps. Once integrated, Spidra will send scraped data directly to a specified Discord channel at your chosen schedule. Let’s get started. *** ## Step 1: Open the Integrations Tab First, navigate to the **Integrations** tab in your Spidra dashboard. Click the **Add Integration** button to begin. Clicking Add Integration in Spidra dashboard ## Step 2: Select Preset and Choose Discord In the **Type** modal that appears, use the dropdown to select the preset you want to connect. Choose Discord as your destination. Click **Next** to proceed. Selecting a preset and choosing Discord as destination ## Step 3: Configure Discord To configure the integration, you’ll need a **Discord Webhook URL**. To get your Webhook URL follow these instructions : * Open your Discord server. * Go to the channel where you want messages to be posted. * Click the gear icon (⚙️) next to the channel name. Selecting a preset and choosing Discord as destination * In the sidebar, go to Integrations → Webhooks. * Click Create Webhook. Create a webhook * Give the webhook a name (e.g., "Spidra Alerts"). Name the webhook * Click Copy Webhook URL. * Click Save Changes. Paste this URL back into the Spidra configuration screen. Confgure the Discord integration ### Optional: Send as File Attachment You can choose to send your scrape result as a file attachment instead of plain text. This is especially useful for: * Large datasets * Preserving formatting (e.g., CSV/Markdown) To enable this, make sure: * You check the "**Attach files**” option to allow your Discord channel permission to upload files. Enabling file attachments in Discord integration Click **Next** to continue. ## Step 4: Set Your Schedule Define how often Spidra should send data to Discord. * Daily * Weekly * Monthly Then set the time of day the integration should run. Timezone: Enter a valid [IANA timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) timezone such as: **Africa/Lagos**, **America/New\_York**, **Europe/London**, etc. Default timezone is **UTC** if not specified. Scheduling the Discord integration run time Click **Next** to proceed. ## Step 5: Name Your Integration Give your integration a descriptive name e.g"*React Job Alerts*". This helps you easily identify it in your integrations dashboard. Naming the integration before final save ## Step 6: Receive Results in Discord Once your scheduled time is reached, head over to the specified Discord channel, your scrape result will be posted automatically. Scrape result sent to Discord channel That’s it! Your Discord integration is live and running on schedule 🎉. # Email Source: https://docs.spidra.io/integrations/email Deliver Spidra scrape results to any email address. Connect a preset, set a schedule, and receive extracted data automatically. This guide shows you how to integrate your **Spidra preset** with **Email**, so that your scraping results are automatically delivered to one or more inboxes at a scheduled time. Follow these steps to get started — setup takes just a few minutes. *** ## Step 1: Open the Integrations Tab Navigate to the **Integrations** tab on your Spidra dashboard and click the **Add Integration** button to begin. Clicking Add Integration in Spidra dashboard ## Step 2: Select Your Preset and Choose Email In the **Type** modal, select the **Preset** you’d like to use from the dropdown, then proceed to choose **Email** as your destination. Click **Next** to proceed. Select preset and choose email delivery ## Step 3: Configure Email Settings In the configuration screen: * Enter the recipient email address. * To send to multiple recipients, use a comma-separated list like this: `team@example.com, manager@example.com, data@yourcompany.com`. **Optionally**: check the **Send as file attachment** box. This is recommended for large data or structured results such as CSV, Markdown, or JSON. If this option is enabled, the scrape result will be sent as a downloadable file (e.g., .csv, .md). Otherwise, the content will appear inline in the email body. Configuring email settings and enabling attachments ## Step 4: Schedule the Integration Set how often the scrape should run: **Daily, Weekly, Monthly**. Then, choose the time of day and set the timezone. By default, Spidra uses **UTC**. If needed, you can use a valid [IANA timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) timezone, such as: * Africa/Lagos * America/New\_York * Asia/Tokyo Email integration schedule and timezone settings Click **Next** to continue. ## Step 5: Name Your Integration Give your integration a meaningful name, this helps you track it later. Examples: "*Jobs alert II*" Naming your email integration ## Step 6: Check Your Inbox for Results Once your integration runs at the scheduled time, the output will be delivered to your inbox. Result of the scrape sent to inbox Important: On your first integration, some email services may flag the message as spam. If that happens, simply mark it as "**Not Spam**" to ensure future results land in your inbox. # Slack Source: https://docs.spidra.io/integrations/slack Deliver Spidra scrape results directly to a Slack channel. Connect a preset, set a schedule, and get structured data posted automatically. This guide walks you through how to integrate your saved [Spidra](https://spidra.io) preset with [Slack](https://slack.com/), enabling you to automatically send scraped data into your Slack workspace in just a few clicks. Spidra offers **two integration methods** for Slack: * **Webhook** – Ideal for sending simple, plain text messages and it is quick to set up. * **Bot Token** – Suitable for advanced workflows and sending file attachments. *** Let’s explore the setup for both methods. ## Step 1: Access the Integrations Page Navigate to the **Integrations** tab on your dashboard and click on the **Add Integration** button. Clicking Add Integration in Spidra dashboard ## Step 2: Select Slack as Integration Type From the preset dropdown, select the saved preset you'd like to connect to Slack. Choose **Slack** as your integration type. Click **Next** to proceed. Clicking Add Integration in Spidra dashboard ## Step 3: Configure Your Slack Integration Spidra supports two ways to connect with Slack. We'll start with the Slack Wehbook: *** ### Method 1: Using a Slack Webhook (Simple, Text-Only) This method is perfect for quickly getting text-based scrape results delivered to a specific channel, it dooesn't support file attachment. #### Steps to Get Your Slack Webhook URL 1. Go to the [Slack Incoming Webhooks page](https://api.slack.com/messaging/webhooks). 2. Click **"Create your Slack app"** (if you don’t already have one). Create Slack App 3. **Create your Slack App**: * Name it something like `"Spidra Notifications"`. * Choose the Slack workspace where you want to install it. * Click **Create App**. App description 4. **Enable Incoming Webhooks**: * In the app dashboard, navigate to **Features → Incoming Webhooks**. Enable Incoming Webhooks * Toggle the switch to **activate webhooks**. 5. **Add the Webhook to a Channel**: * Scroll down and click **“Add New Webhook.”** Enable Incoming Webhooks * Select the channel you want messages sent to. * Click **Allow** to grant access. Add the Webhook to a Channel 6. **Copy Your Webhook URL**: * You’ll be provided a URL like: ``` https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX ``` * Paste this URL into the **Webhook URL field** in Spidra's Slack integration form. Copy Your Webhook URL **Note**: This method supports only plain text — for file attachments, use the Bot Token method. *** ### Method 2: Using a Slack Bot Token (Advanced, File Uploads) If you need to send rich messages or file attachments (e.g., large data exports), we recommend using a Slack Bot Token integration. Since you’ve already created the **Spidra Notifications** Slack app, follow these steps to enable bot functionality. #### Steps to Enable Bot Token in Your Existing Slack App 1. **Go to Your App Settings**: Visit api.slack.com/apps and click on your app: **Spidra Notifications**. Go to Your App Settings 2. **Navigate to OAuth & Permissions**: In the left sidebar, click **OAuth & Permissions**. 3. **Set Scopes for Your Bot Token**: Scroll to Bot Token Scopes and click **Add an OAuth Scope**. Add the following permissions: * `chat:write` → to send messages as the bot. * `files:write` → to upload and send files. * `channels:read` → to access public channels. * `groups:read` → to access private channels (if needed). Set Scopes for Your Bot Token 4. **Install or Reinstall the App**: At the top of the page, click **Install App to Workspace or Reinstall** if already installed. Proceed to authorize the permissions for your workspace. Install or Reinstall the App 5. **Copy Your Bot User OAuth Token**: After installing, scroll up to the OAuth Tokens for Your Workspace section. Copy the token that starts with: ```bash theme={null} xoxb-... ``` Paste this into the Bot Token field in Spidra's Slack integration form. User OAuth Token 6. **Get Your Slack Channel ID**: Spidra also requires your Slack Channel ID to know where to send the data. Here’s how to get it: * Open Slack and go to the desired channel. * Right-click the channel name → Copy Link. You’ll get a URL like: ```bash theme={null} https://slack.com/app_redirect?channel=C1234567890 ``` Copy the value after channel, this is your Channel ID, for example: ```bash theme={null} C1234567890 ``` Paste this ID into the Channel ID field in the integration form. Slack Channel ID *** ## Step 4: Set Up Your Schedule Define how often and when you want the integration to trigger: * **Frequency**: Daily, Weekly, and Monthly. * **Time**: Select the exact time for the scrape job to run. * **Timezone** (default is `UTC`): Use a valid [IANA timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) (e.g., `Africa/Lagos`, `America/New_York`). Click **Next** when you're done. Set Up Your Schedule ## Step 5: Name Your Integration Give your integration a name that reflects its purpose, this helps with managing multiple integrations. **Example**: `Jobs Alert III`, etc. Name Your Integration ## Step 6: Receive Data in Slack Once your scheduled time hits, Spidra will automatically send the scrape results to your selected Slack channel. If you used the Slack Bot Token, you will be able to see the results and also download it. You’ll find the data posted directly in your workspace, exactly how you configured it. Receive Data in Slack *** # Telegram Source: https://docs.spidra.io/integrations/telegram Deliver Spidra scrape results directly to a Telegram chat or channel. Connect a preset, set a schedule, and get data sent automatically. Coming soon! # Webhook Source: https://docs.spidra.io/integrations/webhook Push Spidra scrape results to any webhook URL. Connect a preset, set a schedule, and receive structured JSON in your own endpoint automatically. This guide walks you through integrating your saved preset with a webhook endpoint using [Spidra](https://spidra.io). **Webhooks** are powerful for connecting your scraping workflows with other services like automation platforms, custom apps, or notification tools. Follow the steps below to quickly set up and test a webhook integration. *** ## Step 1: Open the Integrations Tab Go to the **Integrations** tab in the Spidra dashboard and click the **Add integration** button. Clicking Add Integration in Spidra dashboard *** ### Step 2: Choose Integration Type From the preset dropdown, select the preset you want to connect. Then, choose **Webhook** as your destination and click **Next** to continue. Clicking Add Integration in Spidra dashboard *** ## Step 3: Configure Webhook Endpoint Paste your **Webhook Endpoint URL** into the configuration field. This is where Spidra will send the scraped data. We recommend using [Webhook.site](https://webhook.site) for testing purposes. It instantly provides you with a unique temporary URL that displays incoming requests in real time. ### How to Get a Test Webhook URL: 1. Open [https://webhook.site](https://webhook.site) 2. Copy the **unique URL** provided at the top of the page. Clicking Add Integration in Spidra dashboard 3. Paste this URL into Spidra's Webhook input field. Clicking Add Integration in Spidra dashboard *** ## Step 4: Schedule the Integration Set how often you want this webhook to run: * **Daily** * **Weekly** * **Monthly** Specify the **exact time** you want the integration to run and set the **timezone** (default is UTC). Use a valid [IANA timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) string like `Africa/Lagos` or `America/New_York`. Clicking Add Integration in Spidra dashboard *** ### Step 5: Name Your Integration Give your integration a descriptive name, this helps you identify it later in your integration list. Clicking Add Integration in Spidra dashboard *** ## Step 6: View Incoming Payload on Webhook.site Once your scheduled time arrives, navigate to **Webhook.site**. You’ll see the incoming request appear on the page with the full payload. This confirms your integration is working! Clicking Add Integration in Spidra dashboard # Spidra MCP Server Source: https://docs.spidra.io/mcp-server Use Spidra's API through the Model Context Protocol. A [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server that gives AI assistants direct access to Spidra: scraping, batch processing, and crawling the web, all through natural language instead of code. The server is open source and available on [GitHub](https://github.com/spidra-io/spidra-mcp-server). ## Features * Scrape any page into clean markdown or structured JSON * Batch-process up to 50 URLs in parallel, each with its own independent result * Crawl entire sites by describing which links to follow in plain English * Run browser actions before scraping: click, type, scroll, or loop over elements on the page * Route through residential proxies for geo-restricted or bot-protected sites * Streamable HTTP transport, so the server can also run as a network service for tools like n8n * Automatic retries for transient failures, with typed, actionable error messages otherwise ## Installation Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**. Keys start with `spd_`. Spidra's MCP server runs locally today; there is no hosted, keyless endpoint yet. Every setup below uses your own API key. ### Running with npx ```bash theme={null} env SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcp ``` ### Manual installation ```bash theme={null} npm install -g spidra-mcp ``` ### Running on Claude Code ```bash theme={null} claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcp ``` Start a new session, then run `/mcp` to confirm the connection shows as active. ### Running on Cursor 1. Open Cursor Settings 2. Go to **Features** > **MCP Servers** 3. Click **+ Add new global MCP server** 4. Enter the following, replacing the placeholder key: ```json theme={null} { "mcpServers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "YOUR-API-KEY" } } } } ``` You can also put this in a `.cursor/mcp.json` file inside a single project if you only want Spidra available there. ### Running on Windsurf Add this to `~/.codeium/windsurf/mcp_config.json`: ```json theme={null} { "mcpServers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "YOUR_API_KEY" } } } } ``` ### Running on VS Code Press `Ctrl + Shift + P` (or `Cmd + Shift + P` on Mac), type `Preferences: Open User Settings (JSON)`, and add: ```json theme={null} { "mcp": { "servers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "YOUR_API_KEY" } } } } } ``` To share the setup with your team instead, put the same `servers` block in a `.vscode/mcp.json` file in your repository, and use a `promptString` input for the key so it never gets committed. ### Running on Claude Desktop Add this to the Claude config file (**Settings** > **Developer** > **Edit Config**): ```json theme={null} { "mcpServers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "YOUR_API_KEY_HERE" } } } } ``` Quit and reopen Claude Desktop afterward. If you see a `spawn npx ENOENT` error, Node.js is not installed or not on your system PATH; install it from [nodejs.org](https://nodejs.org) and fully restart Claude Desktop. ### Running with streamable HTTP mode To run the server over HTTP instead of the default stdio transport: ```bash theme={null} env HTTP_STREAMABLE_SERVER=true SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcp ``` Use the URL `http://localhost:3000/mcp`. ### Running on n8n n8n's **AI Agent** node can call Spidra through its **MCP Client Tool** node, connecting over the streamable HTTP transport above. 1. Start the server in HTTP mode, as shown just above 2. In your n8n workflow, add an **AI Agent** node 3. In its configuration, add a new **Tool** and choose **MCP Client Tool** 4. Set the **Endpoint** to `http://localhost:3000/mcp` 5. Set **Server Transport** to **HTTP Streamable** 6. For **Tools to include**, choose **All** to expose every Spidra tool, or pick specific ones If n8n is running somewhere else, for example inside Docker, run the MCP server on a host and port n8n can actually reach, and adjust the endpoint to match. ## Configuration ### Environment variables * `SPIDRA_API_KEY`: your Spidra API key. Required, unless every request supplies its own key over HTTP (see below). * `SPIDRA_API_URL` (optional): override the base API URL, for staging or self-hosted setups. * `HTTP_STREAMABLE_SERVER` (optional): set to `true` to run the streamable HTTP transport instead of stdio. * `PORT` / `HOST` (optional): bind address for the HTTP transport. Defaults are `3000` and `localhost`. On the HTTP transport, the API key can also be sent per request using an `X-Spidra-API-Key` header, or an `Authorization: Bearer` header. This is useful when a single running server instance needs to serve more than one user. ## Available Tools The server exposes 12 tools. The table below is a quick reference; each tool is described in more detail after it. | Tool | What it does | | ---------------------------- | ---------------------------------------------------------- | | `spidra_scrape` | Scrape 1 to 3 URLs and wait for the extracted result | | `spidra_check_scrape_status` | Check a scrape that outlived its wait window | | `spidra_batch_scrape` | Scrape 2 to 50 URLs in parallel, each with its own result | | `spidra_check_batch_status` | Progress and per-URL results for a batch | | `spidra_cancel_batch` | Cancel a batch that is no longer needed | | `spidra_crawl` | Discover and process pages starting from one URL | | `spidra_check_crawl_status` | Progress, then full results, for a crawl | | `spidra_crawl_pages` | Per-page results with raw HTML and markdown download links | | `spidra_crawl_extract` | Ask a new question of an already-completed crawl | | `spidra_cancel_crawl` | Cancel a crawl that is no longer needed | | `spidra_scrape_logs` | Look up past jobs and their outputs | | `spidra_usage` | Check credit and request usage | ### 1. Scrape Tool (`spidra_scrape`) Scrapes 1 to 3 URLs and extracts content with AI, waiting for the result before returning it. With more than one URL, their content is merged and the AI produces one combined answer across all of them, which makes this the right tool for comparing pages rather than getting separate results for each. ```json theme={null} { "name": "spidra_scrape", "arguments": { "urls": ["https://example.com/product"], "prompt": "Extract the product name, price, and description", "output": "json", "schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "description": { "type": "string" } }, "required": ["name", "price"] } } } ``` **Returns:** the extracted `content`, per-page raw data, any `screenshots`, and token `stats`. ### 2. Check Scrape Status (`spidra_check_scrape_status`) Checks a scrape job by ID. Only needed when a scrape outlives its wait window; the timeout error includes the job ID to use here. ```json theme={null} { "name": "spidra_check_scrape_status", "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" } } ``` ### 3. Batch Scrape Tool (`spidra_batch_scrape`) Submits 2 to 50 URLs processed in parallel with the same prompt or schema. Unlike `spidra_scrape`, every URL is handled independently and gets its own result. Returns immediately with a `batchId` for polling. ```json theme={null} { "name": "spidra_batch_scrape", "arguments": { "urls": ["https://shop.example.com/product/1", "https://shop.example.com/product/2"], "prompt": "Extract the product name, price, and star rating", "output": "json" } } ``` **Returns:** `{ "batchId": "...", "total": 2 }`. ### 4. Check Batch Status (`spidra_check_batch_status`) Returns batch progress (`completedCount`, `failedCount`) and per-URL results for every finished item. ```json theme={null} { "name": "spidra_check_batch_status", "arguments": { "batchId": "batch-uuid-here" } } ``` ### 5. Cancel Batch (`spidra_cancel_batch`) Cancels a running batch. Finished items keep their results; credits for the unprocessed portion are refunded. ```json theme={null} { "name": "spidra_cancel_batch", "arguments": { "batchId": "batch-uuid-here" } } ``` ### 6. Crawl Tool (`spidra_crawl`) Crawls a website from one starting URL, following links according to a plain-English instruction, and optionally extracting structured data from every page. Returns immediately with a `jobId`. ```json theme={null} { "name": "spidra_crawl", "arguments": { "baseUrl": "https://example.com/blog", "crawlInstruction": "Follow blog post links only, skip tag and category pages", "transformInstruction": "Extract the title, author, and publish date", "maxPages": 10 } } ``` **Returns:** `{ "jobId": "..." }`. ### 7. Check Crawl Status (`spidra_check_crawl_status`) Returns crawl progress while running, and every page's extracted data once completed. ```json theme={null} { "name": "spidra_check_crawl_status", "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" } } ``` ### 8. Crawl Pages (`spidra_crawl_pages`) Returns per-page results with signed download URLs for each page's raw HTML and markdown. The links expire after 1 hour. ```json theme={null} { "name": "spidra_crawl_pages", "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" } } ``` ### 9. Crawl Extract (`spidra_crawl_extract`) Runs a new extraction instruction over an already-completed crawl without fetching any pages again. Only AI token credits are charged, so this is the cheap way to ask a second question of a site you already crawled. ```json theme={null} { "name": "spidra_crawl_extract", "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000", "transformInstruction": "List the main topics each post covers" } } ``` ### 10. Cancel Crawl (`spidra_cancel_crawl`) Cancels a running crawl. Pages already processed are kept and retrievable with `spidra_crawl_pages`; credits for unprocessed pages are refunded. ```json theme={null} { "name": "spidra_cancel_crawl", "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" } } ``` ### 11. Scrape Logs (`spidra_scrape_logs`) Lists past jobs: what ran, whether it succeeded, and how many credits it used. Pass a `uuid` to fetch one job's full output. ```json theme={null} { "name": "spidra_scrape_logs", "arguments": { "status": "failed", "limit": 10 } } ``` ### 12. Usage (`spidra_usage`) Reports request, credit, and token usage by day or week. ```json theme={null} { "name": "spidra_usage", "arguments": { "range": "7d" } } ``` For the full parameter tables, tool-choice guidance, and troubleshooting steps, see the [complete MCP reference](/sdks/mcp). ## Error Handling Every API error comes back as a tool error, not a crash. The error text carries retry guidance: rate limits say how long to wait, validation errors say exactly what to fix, and permanent failures say not to retry with the same input. ```json theme={null} { "content": [ { "type": "text", "text": "Spidra API error 404 (JOB_NOT_FOUND): Scrape not found. Do not retry with the same input." } ], "isError": true } ``` ## Development The server is open source at [github.com/spidra-io/spidra-mcp-server](https://github.com/spidra-io/spidra-mcp-server). ```bash theme={null} git clone https://github.com/spidra-io/spidra-mcp-server.git cd spidra-mcp-server npm install npm run build # bundles src/index.ts to dist/index.js npm test # builds, then runs the test suite against a fake API npm run typecheck # type-checks without emitting ``` The test suite spawns the built server as a real process and talks to it over stdio JSON-RPC, the same way an MCP client would, against a fake Spidra API so no credits are spent. Pull requests are welcome. Requires Node.js 20 or newer. MIT licensed. Parameter tables, tool-choice guidance, and troubleshooting. Prefer code over chat? Browse the official SDKs. # Quickstart Source: https://docs.spidra.io/quickstart Make your first scrape in under 2 minutes. ## Step 1: Create your account and get an API key [Sign up](https://app.spidra.io/login) with your email, Google, or GitHub account. Sign up or log in using Google or GitHub Once you're in, go to **Settings → API Keys** and create your first key. Store your key in an environment variable called `SPIDRA_API_KEY`. Never commit it to source control. You get 300 free credits when you sign up. No credit card needed. Now pick how you want to work. ## Step 2: Open the Playground and add your URL Click **Playground** in the sidebar. Paste a URL into the Target URL field, then write what you want to extract in plain English. Navigating to the Playground Try this to start: ```bash theme={null} URL: https://remoteok.com/remote-react-jobs Prompt: Extract all job listings on this page. For each job return the title, company name, link to the post, and salary if shown. Return as JSON. ``` Entering a target URL and extraction prompt ## Step 3: Set your options and run Choose **JSON** as your output format. Then toggle any options you need: * **Extract only content** removes navbars, footers, and boilerplate so the AI focuses on what matters. * **Stealth mode** routes the request through a residential proxy, useful on sites that block scrapers. * **Screenshot** captures the page at scrape time, handy for debugging. Output format and options Hit **Start Scraping**. Spidra opens the page in a real browser, runs your prompt through the AI, and returns the extracted results on the right side of the screen. Extracted results shown in the results panel ## Step 4: Save it as a preset Click **Save as Preset** and give it a name. A preset stores your URL, prompt, and all settings so you can rerun it with one click, put it on a schedule, or connect it to Slack, Discord, or a webhook. This is how recurring workflows are built in Spidra: preset, then integration, then schedule. Set it up once and the data flows automatically. Click, scroll, and type before extracting. Works on any dynamic page. Guarantee exact field names and types with a JSON schema. Deliver data to Slack, Airtable, or webhooks automatically. Find pre-built presets for common scraping jobs. Prefer to watch? [Three-minute video walkthrough →](https://www.youtube.com/watch?v=mjewZ4qU9Mk) ## Step 2: Install an SDK Pick the language your project uses. They all wrap the same API. If you'd rather go direct with curl or HTTP, skip this step — there's nothing to install. ```bash Node.js theme={null} npm install spidra ``` ```bash Python theme={null} pip install spidra ``` ```bash PHP theme={null} composer require spidra/spidra-php ``` ```bash Go theme={null} go get github.com/spidra-io/spidra-go ``` ```bash Ruby theme={null} gem install spidra ``` ```bash Elixir theme={null} # Add to mix.exs deps, then: mix deps.get ``` ```bash .NET theme={null} dotnet add package Spidra ``` ```bash Swift theme={null} # In Package.swift: .package(url: "https://github.com/spidra-io/spidra-swift.git", from: "1.0.0") ``` ```bash Java theme={null} # Gradle: implementation 'io.spidra:spidra-java-sdk:0.1.0' # Maven: # io.spidraspidra-java-sdk0.1.0 ``` ```bash Rust theme={null} cargo add spidra ``` ## Step 3: Make your first scrape Spidra jobs are async. The `run()` method handles polling for you and returns when the job completes. If you need more control, use `submit()` to queue the job and `get()` to check status yourself. ```bash curl theme={null} # 1. Submit the job curl -X POST https://api.spidra.io/api/scrape \ -H "Authorization: Bearer $SPIDRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [{"url": "https://remoteok.com/remote-react-jobs"}], "prompt": "Extract all job listings. For each one return the title, company, and salary if shown.", "output": "json" }' # Response: {"jobId": "abc123", "status": "queued"} # 2. Poll until completed curl https://api.spidra.io/api/scrape/abc123 \ -H "Authorization: Bearer $SPIDRA_API_KEY" ``` ```typescript Node.js theme={null} import { SpidraClient } from 'spidra'; const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY }); const job = await spidra.scrape.run({ urls: [{ url: 'https://remoteok.com/remote-react-jobs' }], prompt: 'Extract all job listings. For each one return the title, company, and salary if shown.', output: 'json', }); console.log(job.result.content); ``` ```python Python theme={null} import asyncio from spidra import AsyncSpidra, ScrapeParams, ScrapeUrl async def main(): client = AsyncSpidra(api_key="spd_YOUR_API_KEY") result = await client.scrape(ScrapeParams( urls=[ScrapeUrl(url="https://remoteok.com/remote-react-jobs")], prompt="Extract all job listings. For each one return the title, company, and salary if shown.", output="json", )) print(result.content) asyncio.run(main()) ``` ```php PHP theme={null} use Spidra\SpidraClient; $spidra = new SpidraClient(getenv('SPIDRA_API_KEY')); $job = $spidra->scrape->run([ 'urls' => [['url' => 'https://remoteok.com/remote-react-jobs']], 'prompt' => 'Extract all job listings. For each one return the title, company, and salary if shown.', 'output' => 'json', ]); print_r($job['content']); ``` ```go Go theme={null} package main import ( "context" "fmt" "os" spidra "github.com/spidra-io/spidra-go" ) func main() { client := spidra.New(os.Getenv("SPIDRA_API_KEY")) job, err := client.Scrape.Run(context.Background(), spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{{URL: "https://remoteok.com/remote-react-jobs"}}, Prompt: "Extract all job listings. For each one return the title, company, and salary if shown.", Output: "json", }) if err != nil { panic(err) } fmt.Println(job.Result.Content) } ``` ```ruby Ruby theme={null} require "spidra" client = Spidra.new(ENV["SPIDRA_API_KEY"]) job = client.scrape.run( urls: [{ url: "https://remoteok.com/remote-react-jobs" }], prompt: "Extract all job listings. For each one return the title, company, and salary if shown.", output: "json" ) puts job["result"]["content"] ``` ```elixir Elixir theme={null} config = Spidra.Config.new(api_key: System.get_env("SPIDRA_API_KEY")) {:ok, job} = Spidra.Scrape.run(config, %{ urls: [%{url: "https://remoteok.com/remote-react-jobs"}], prompt: "Extract all job listings. For each one return the title, company, and salary if shown.", output: "json" }) IO.inspect(job["result"]["content"]) ``` ```csharp .NET theme={null} using Spidra; using Spidra.Types.Scrape; var client = new SpidraClient(Environment.GetEnvironmentVariable("SPIDRA_API_KEY")!); var job = await client.Scrape.RunAsync(new ScrapeParams { Urls = [new ScrapeUrl("https://remoteok.com/remote-react-jobs")], Prompt = "Extract all job listings. For each one return the title, company, and salary if shown.", Output = OutputFormat.Json }); Console.WriteLine(job.Result.Content); ``` ```swift Swift theme={null} import Spidra let spidra = SpidraClient(apiKey: ProcessInfo.processInfo.environment["SPIDRA_API_KEY"]!) let params = ScrapeParams( urls: [ScrapeUrl(url: "https://remoteok.com/remote-react-jobs")], prompt: "Extract all job listings. For each one return the title, company, and salary if shown.", output: "json" ) let job = try await spidra.scrape.run(params) print(job.result?.content?.value ?? "No data") ``` ```java Java theme={null} import io.spidra.sdk.SpidraClient; import io.spidra.sdk.model.scrape.ScrapeParams; import io.spidra.sdk.model.scrape.ScrapeJob; SpidraClient client = new SpidraClient(System.getenv("SPIDRA_API_KEY")); ScrapeParams params = ScrapeParams.builder() .url("https://remoteok.com/remote-react-jobs") .prompt("Extract all job listings. For each one return the title, company, and salary if shown.") .outputFormat("json") .build(); ScrapeJob job = client.scrape().run(params).join(); System.out.println(job.getResult().getContent()); ``` ```rust Rust theme={null} use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").unwrap()); let mut params = ScrapeParams::new("https://remoteok.com/remote-react-jobs"); params.output_format = Some(OutputFormat::Json); params.prompt = Some( "Extract all job listings. For each one return the title, company, and salary if shown." .to_string(), ); let result = client.scrape().run(¶ms).await?; println!("{}", result.content.unwrap_or_default()); Ok(()) } ``` The result comes back structured, ready to use: ```json theme={null} { "jobs": [ { "title": "Senior React Engineer", "company": "Acme Corp", "salary": "$140,000 – $180,000", "link": "https://remoteok.com/jobs/123456" }, { "title": "React Frontend Developer", "company": "Startup Inc", "salary": null, "link": "https://remoteok.com/jobs/654321" } ] } ``` ## Step 4: Follow links with forEach The basic scrape reads a single page. `forEach` is what you use when you need to click into each item and pull detail from the destination page. This is where most scrapers fall short — Spidra handles it natively. ```bash curl theme={null} curl -X POST https://api.spidra.io/api/scrape \ -H "Authorization: Bearer $SPIDRA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "urls": [ { "url": "https://remoteok.com/remote-react-jobs", "actions": [ { "type": "forEach", "observe": "Find all job listing rows on the page", "mode": "navigate", "maxItems": 10, "waitAfterClick": 1000, "itemPrompt": "Extract the full job description, required skills, and salary range. Return as JSON." } ] } ], "prompt": "Return a clean JSON array of all jobs with their full details.", "output": "json" }' ``` ```typescript Node.js theme={null} const job = await spidra.scrape.run({ urls: [ { url: 'https://remoteok.com/remote-react-jobs', actions: [ { type: 'forEach', observe: 'Find all job listing rows on the page', mode: 'navigate', maxItems: 10, waitAfterClick: 1000, itemPrompt: 'Extract the full job description, required skills, and salary range. Return as JSON.', }, ], }, ], prompt: 'Return a clean JSON array of all jobs with their full details.', output: 'json', }); ``` ```python Python theme={null} from spidra import BrowserAction result = await client.scrape(ScrapeParams( urls=[ ScrapeUrl( url="https://remoteok.com/remote-react-jobs", actions=[ BrowserAction( type="forEach", observe="Find all job listing rows on the page", mode="navigate", max_items=10, wait_after_click=1000, item_prompt="Extract the full job description, required skills, and salary range. Return as JSON.", ), ], ), ], prompt="Return a clean JSON array of all jobs with their full details.", output="json", )) ``` ```php PHP theme={null} $job = $spidra->scrape->run([ 'urls' => [ [ 'url' => 'https://remoteok.com/remote-react-jobs', 'actions' => [ [ 'type' => 'forEach', 'observe' => 'Find all job listing rows on the page', 'mode' => 'navigate', 'maxItems' => 10, 'waitAfterClick' => 1000, 'itemPrompt' => 'Extract the full job description, required skills, and salary range. Return as JSON.', ], ], ], ], 'prompt' => 'Return a clean JSON array of all jobs with their full details.', 'output' => 'json', ]); ``` ```go Go theme={null} job, err := client.Scrape.Run(context.Background(), spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{ { URL: "https://remoteok.com/remote-react-jobs", Actions: []spidra.BrowserAction{ { Type: "forEach", Observe: "Find all job listing rows on the page", Mode: "navigate", MaxItems: 10, WaitAfterClick: 1000, ItemPrompt: "Extract the full job description, required skills, and salary range. Return as JSON.", }, }, }, }, Prompt: "Return a clean JSON array of all jobs with their full details.", Output: "json", }) ``` ```ruby Ruby theme={null} job = client.scrape.run( urls: [ { url: "https://remoteok.com/remote-react-jobs", actions: [ { type: "forEach", observe: "Find all job listing rows on the page", mode: "navigate", maxItems: 10, waitAfterClick: 1000, itemPrompt: "Extract the full job description, required skills, and salary range. Return as JSON." } ] } ], prompt: "Return a clean JSON array of all jobs with their full details.", output: "json" ) ``` ```elixir Elixir theme={null} {:ok, job} = Spidra.Scrape.run(config, %{ urls: [ %{ url: "https://remoteok.com/remote-react-jobs", actions: [ %{ type: "forEach", observe: "Find all job listing rows on the page", mode: "navigate", max_items: 10, wait_after_click: 1000, item_prompt: "Extract the full job description, required skills, and salary range. Return as JSON." } ] } ], prompt: "Return a clean JSON array of all jobs with their full details.", output: "json" }) ``` ```csharp .NET theme={null} using Spidra; using Spidra.Types.Scrape; using System.Text.Json; // Browser actions are sent as raw JSON alongside the URL. // Build the request body as an anonymous object and post via RunAsync. var job = await client.Scrape.RunAsync(new ScrapeParams { Urls = [new ScrapeUrl("https://remoteok.com/remote-react-jobs")], Prompt = "Return a clean JSON array of all jobs with their full details.", Output = OutputFormat.Json, // forEach actions are included via the raw JSON body through // the API — see the curl tab for the exact payload shape. }); Console.WriteLine(job.Result.Content); ``` ```swift Swift theme={null} import Spidra let forEachAction = BrowserAction.forEach( observe: "Find all job listing rows on the page", mode: "navigate", captureSelector: nil, maxItems: 10, itemPrompt: "Extract the full job description, required skills, and salary range. Return as JSON.", waitAfterClick: 1000, actions: nil, pagination: nil ) let url = ScrapeUrl( url: "https://remoteok.com/remote-react-jobs", actions: [forEachAction] ) let params = ScrapeParams( urls: [url], prompt: "Return a clean JSON array of all jobs with their full details.", output: "json" ) let job = try await spidra.scrape.run(params) print(job.result?.content?.value ?? "No data") ``` ```java Java theme={null} import io.spidra.sdk.model.scrape.BrowserAction; import java.util.List; // The Java SDK BrowserAction builder supports type, selector, value, // waitMs, and url. For forEach, pass the action type and supply // the observe/mode/itemPrompt fields via the REST API (curl tab) // or use a custom ObjectNode to extend the payload. ScrapeParams params = ScrapeParams.builder() .url("https://remoteok.com/remote-react-jobs") .browserActions(List.of( BrowserAction.builder() .type("forEach") .value("Find all job listing rows on the page") .build() )) .prompt("Return a clean JSON array of all jobs with their full details.") .outputFormat("json") .build(); ScrapeJob job = client.scrape().run(params).join(); System.out.println(job.getResult().getContent()); ``` ```rust Rust theme={null} // The Rust SDK's typed BrowserAction enum does not yet include a // forEach variant. Use the curl example above to send forEach // actions directly via the REST API. use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").unwrap()); let mut params = ScrapeParams::new("https://remoteok.com/remote-react-jobs"); params.output_format = Some(OutputFormat::Json); params.prompt = Some("Return a clean JSON array of all jobs with their full details.".to_string()); let result = client.scrape().run(¶ms).await?; println!("{}", result.content.unwrap_or_default()); Ok(()) } ``` Each matched element gets its own AI extraction pass. The results come back as a structured array. Read the full [browser actions guide](/features/actions) to see what else is possible. The full forEach reference, plus click, scroll, type, and pagination. Use a JSON schema to enforce exact field names and types on every response. Submit up to 50 URLs in one request and process them all in parallel. Every endpoint with request and response examples if you're going direct. # .NET Source: https://docs.spidra.io/sdks/dotnet Official .NET SDK for Spidra — AI-powered web scraping with proxy rotation and CAPTCHA handling. The official .NET SDK for Spidra lets you extract structured data from any website by describing what you want in plain English. It handles JavaScript rendering, anti-bot bypass, and CAPTCHA solving as a managed API, so your code stays focused on the data. ## Installation ```bash theme={null} dotnet add package Spidra ``` Requires .NET 8 or later. Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Keep your API key out of source control — read it from an environment variable or a secrets manager. *** ## Getting started All requests require an API key. Pass it to the client at initialization: ```csharp theme={null} var client = new SpidraClient(Environment.GetEnvironmentVariable("SPIDRA_API_KEY")!); ``` ## Quick start ```csharp theme={null} using Spidra; using Spidra.Types.Scrape; var client = new SpidraClient(Environment.GetEnvironmentVariable("SPIDRA_API_KEY")!); var job = await client.Scrape.RunAsync(new ScrapeParams { Urls = [new ScrapeUrl("https://news.ycombinator.com")], Prompt = "List the top 5 stories with title, points, and comment count", UseProxy = true }); Console.WriteLine(job.Result.Content); ``` `RunAsync` submits the job and polls until it completes, then returns the result. *** ## Scraping ### RunAsync — submit and wait ```csharp theme={null} var job = await client.Scrape.RunAsync(new ScrapeParams { Urls = [new ScrapeUrl("https://example.com/pricing")], Prompt = "Extract all pricing plans with name, price, and included features", Output = OutputFormat.Json }); Console.WriteLine(job.Result.Content); ``` **Parameters** | Property | Type | Description | | -------------------- | -------------- | ----------------------------------------------------------- | | `Urls` | `ScrapeUrl[]` | Up to 3 URLs, each with optional per-URL browser actions | | `Prompt` | `string` | AI extraction instruction | | `Output` | `OutputFormat` | `OutputFormat.Markdown` (default) or `OutputFormat.Json` | | `Schema` | `JsonElement?` | JSON Schema for guaranteed output shape | | `UseProxy` | `bool` | Route through a residential proxy | | `ProxyCountry` | `string?` | Two-letter country code, e.g. `"us"`, `"de"`, `"jp"` | | `ExtractContentOnly` | `bool` | Strip navigation, ads, and boilerplate before AI extraction | | `Screenshot` | `bool` | Capture a screenshot of the page | | `FullPageScreenshot` | `bool` | Capture a full-page (scrolled) screenshot | | `Cookies` | `string?` | Raw `Cookie` header string for authenticated pages | ### SubmitAsync + GetAsync — manual control If you need to track progress yourself, use `SubmitAsync` and `GetAsync` directly: ```csharp theme={null} var job = await client.Scrape.SubmitAsync(new ScrapeParams { Urls = [new ScrapeUrl("https://example.com")], Prompt = "Extract the main heading" }); Console.WriteLine($"Job submitted: {job.JobId}"); while (job.Status is not ("completed" or "failed")) { await Task.Delay(TimeSpan.FromSeconds(2)); job = await client.Scrape.GetAsync(job.JobId); Console.WriteLine($"Status: {job.Status}"); } ``` **Job statuses:** `waiting` · `active` · `completed` · `failed` ### Structured output Pass a JSON schema to get a typed, deserializable result instead of raw text: ```csharp theme={null} using System.Text.Json; var job = await client.Scrape.RunAsync(new ScrapeParams { Urls = [new ScrapeUrl("https://jobs.example.com/senior-engineer")], Prompt = "Extract the job title, company, location, and required skills", Output = OutputFormat.Json, Schema = JsonSerializer.SerializeToElement(new { type = "object", required = new[] { "title", "company" }, properties = new { title = new { type = "string" }, company = new { type = "string" }, location = new { type = new[] { "string", "null" } }, skills = new { type = "array", items = new { type = "string" } } } }) }); var listing = job.Result.Content.Deserialize(new JsonSerializerOptions { PropertyNameCaseInsensitive = true }); Console.WriteLine($"{listing!.Title} at {listing.Company}"); Console.WriteLine($"Skills: {string.Join(", ", listing.Skills)}"); record JobListing(string Title, string Company, string? Location, List Skills); ``` Fields in `required` always appear in the response (as `null` if the data is not found). Optional fields are omitted when unavailable. *** ## Batch scraping Process up to 50 URLs in one call. All URLs are processed in parallel. ```csharp theme={null} using Spidra.Types.Batch; var batch = await client.Batch.RunAsync(new BatchScrapeParams { Urls = [ "https://competitor-a.com/pricing", "https://competitor-b.com/pricing", "https://competitor-c.com/pricing" ], Prompt = "Extract all pricing plans with name and monthly price", Output = OutputFormat.Json, UseProxy = true }); var succeeded = batch.Items.Where(i => i.Status == "completed").ToList(); Console.WriteLine($"{succeeded.Count}/{batch.Items.Count} succeeded"); foreach (var item in succeeded) { Console.WriteLine($"{item.Url}: {item.Result}"); } ``` **Item statuses:** `pending` · `running` · `completed` · `failed` **Batch statuses:** `pending` · `running` · `completed` · `failed` · `cancelled` ### Retry failed items ```csharp theme={null} if (batch.Items.Any(i => i.Status == "failed")) await client.Batch.RetryAsync(batch.BatchId); ``` *** ## Crawling Crawl an entire site and extract structured data from each page. ```csharp theme={null} using Spidra.Types.Crawl; var job = await client.Crawl.RunAsync(new CrawlParams { BaseUrl = "https://example.com/blog", CrawlInstruction = "Find all blog posts published in 2024", TransformInstruction = "Extract title, author, publish date, and summary", MaxPages = 50, UseProxy = true }); foreach (var page in job.Result) { Console.WriteLine($"{page.Url}: {page.Data}"); } ``` **Parameters** | Property | Type | Description | | ---------------------- | --------- | -------------------------------------------------- | | `BaseUrl` | `string` | Starting URL for the crawl | | `CrawlInstruction` | `string` | Which links to follow and which to skip | | `TransformInstruction` | `string` | What to extract from each page | | `MaxPages` | `int` | Maximum number of pages to crawl | | `UseProxy` | `bool` | Route through a residential proxy | | `ProxyCountry` | `string?` | Two-letter country code, e.g. `"us"` | | `Cookies` | `string?` | Raw `Cookie` header string for authenticated sites | *** ## Error handling All exceptions inherit from `SpidraException`. | Exception | When | | ------------------------------------ | -------------------------------- | | `SpidraAuthenticationException` | 401 — invalid or missing API key | | `SpidraInsufficientCreditsException` | 402 — not enough credits | | `SpidraRateLimitException` | 429 — rate limit exceeded | | `SpidraServerException` | 5xx — server-side error | ```csharp theme={null} using Spidra.Exceptions; try { var job = await client.Scrape.RunAsync(scrapeParams); return job.Result.Content; } catch (SpidraAuthenticationException) { logger.LogError("Invalid API key. Check your SPIDRA_API_KEY."); throw; } catch (SpidraInsufficientCreditsException) { logger.LogWarning("Out of scraping credits. Upgrade at spidra.io."); throw; } catch (SpidraRateLimitException ex) { await Task.Delay(ex.RetryAfter ?? TimeSpan.FromSeconds(5)); // retry... } catch (SpidraServerException) { logger.LogError("Spidra server error."); throw; } ``` `SpidraRateLimitException.RetryAfter` contains the server-suggested wait time when available. Official Elixir SDK — idiomatic pattern matching, OTP-ready, works with Phoenix and plain Mix projects. Official Swift SDK — async/await native, works on iOS, macOS, tvOS, watchOS, and server-side Swift. # Elixir Source: https://docs.spidra.io/sdks/elixir Official Elixir SDK for Spidra — scrape pages, run browser actions, batch-process URLs, and crawl entire sites. The official Elixir SDK for Spidra wraps the Spidra API so you're not writing raw HTTP calls and polling loops yourself. It handles job submission, status polling, retry logic, and error mapping. All results come back as structured data ready to feed into your LLM pipelines or store directly. ## Installation Add `spidra` to your list of dependencies in `mix.exs`: ```elixir theme={null} def deps do [ {:spidra, "~> 0.1.0"} ] end ``` Then run `mix deps.get` in your terminal. Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Store it as an environment variable — never hardcode it in source. ## Requirements * Elixir 1.14 or later * A Spidra API key ([sign up free](https://spidra.io)) *** ## Getting started ```elixir theme={null} # Initialize your configuration config = Spidra.Config.new(api_key: "spd_YOUR_API_KEY") ``` From here you access everything through `Spidra.Scrape`, `Spidra.Batch`, `Spidra.Crawl`, `Spidra.Logs`, and `Spidra.Usage`. ## Scraping All scrape jobs run asynchronously on the Spidra platform. `Spidra.Scrape.run/3` submits a job and polls until it finishes. If you need more control, use `submit/2` and `get/2` directly. Up to 3 URLs can be passed per request and they are processed in parallel. ### Basic scrape Submit a scrape job and wait for results. ```elixir theme={null} {:ok, job} = Spidra.Scrape.run(config, %{ urls: [%{url: "https://example.com/pricing"}], prompt: "Extract all pricing plans with name, price, and included features", output: "json" }) IO.inspect(job["result"]["content"]) # "{ \"plans\": [{ \"name\": \"Starter\", \"price\": \"$9/mo\", \"features\": [...] }, ...] }" ``` **Parameters** | Parameter | Type | Description | | ---------------------- | ------- | -------------------------------------------------------------------- | | `urls` | list | Up to 3 URL maps. Each takes a `url` key and optional `actions` list | | `prompt` | string | AI extraction instruction | | `output` | string | `"markdown"` (default) or `"json"` | | `schema` | map | JSON Schema for guaranteed output shape (use with `output: "json"`) | | `use_proxy` | boolean | Route through a residential proxy | | `proxy_country` | string | Two-letter country code, e.g. `"us"`, `"de"`, `"jp"` | | `extract_content_only` | boolean | Strip navigation, ads, and boilerplate before AI extraction | | `screenshot` | boolean | Capture a screenshot of the page | | `full_page_screenshot` | boolean | Capture a full-page (scrolled) screenshot | | `cookies` | string | Raw `Cookie` header string for authenticated pages | ### Fire-and-forget approach Use `submit/2` and `get/2` when you want to manage polling yourself. ```elixir theme={null} # Submit a job and get the job_id immediately {:ok, %{"jobId" => job_id}} = Spidra.Scrape.submit(config, %{ urls: [%{url: "https://example.com"}], prompt: "Extract the main headline" }) # Check status at any point {:ok, status} = Spidra.Scrape.get(config, job_id) case status["status"] do "completed" -> IO.inspect(status["result"]["content"]) "failed" -> IO.inspect(status["error"]) _ -> IO.puts("Job is still pending...") end ``` **Job statuses:** `waiting` · `active` · `completed` · `failed` ### Structured JSON output Pass a `schema` to enforce an exact output shape. Missing fields come back as `null` rather than hallucinated values. ```elixir theme={null} {:ok, job} = Spidra.Scrape.run(config, %{ urls: [%{url: "https://jobs.example.com/senior-engineer"}], prompt: "Extract the job listing details", output: "json", schema: %{ "type" => "object", "required" => ["title", "company", "remote"], "properties" => %{ "title" => %{"type" => "string"}, "company" => %{"type" => "string"}, "remote" => %{"type" => ["boolean", "null"]}, "salary_min" => %{"type" => ["number", "null"]}, "salary_max" => %{"type" => ["number", "null"]}, "skills" => %{"type" => "array", "items" => %{"type" => "string"}} } } }) ``` ### Geo-targeted scraping Pass `use_proxy: true` and a `proxy_country` code to route the request through a specific country. Useful for geo-restricted content or localized pricing. ```elixir theme={null} {:ok, job} = Spidra.Scrape.run(config, %{ urls: [%{url: "https://www.amazon.de/gp/bestsellers"}], prompt: "List the top 10 products with name and price", use_proxy: true, proxy_country: "de" }) ``` Supported country codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, `sg`, `es`, `it`, `mx`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing. ### Authenticated pages Pass cookies as a string to scrape pages that require a login session. ```elixir theme={null} {:ok, job} = Spidra.Scrape.run(config, %{ urls: [%{url: "https://app.example.com/dashboard"}], prompt: "Extract the monthly revenue and active user count", cookies: "session=abc123; auth_token=xyz789" }) ``` ### Browser actions Actions let you interact with the page before the scrape runs. They execute in order, and the scrape happens after all actions complete. ```elixir theme={null} {:ok, job} = Spidra.Scrape.run(config, %{ urls: [ %{ url: "https://example.com/products", actions: [ %{type: "click", selector: "#accept-cookies"}, %{type: "wait", duration: 1000}, %{type: "scroll", to: "80%"} ] } ], prompt: "Extract all product names and prices" }) ``` **Available actions** | Action | Required fields | Description | | --------- | --------------------- | ---------------------------------------------------- | | `click` | `selector` or `value` | Click a button, link, or any element | | `type` | `selector`, `value` | Type text into an input or textarea | | `check` | `selector` or `value` | Check a checkbox | | `uncheck` | `selector` or `value` | Uncheck a checkbox | | `wait` | `duration` (ms) | Pause for a set number of milliseconds | | `scroll` | `to` (`0–100%`) | Scroll the page to a percentage of its height | | `forEach` | `observe` | Loop over every matched element and process each one | Use `selector` for a CSS selector or XPath. Use `value` for plain English — Spidra locates the element using AI. *** ## Batch scraping Submit up to 50 URLs in a single request. All URLs are processed in parallel. Each URL is a plain string. ```elixir theme={null} {:ok, batch} = Spidra.Batch.run(config, %{ urls: [ "https://shop.example.com/product/1", "https://shop.example.com/product/2", "https://shop.example.com/product/3" ], prompt: "Extract product name, price, and availability", output: "json", use_proxy: true }) for item <- batch["items"] do case item["status"] do "completed" -> IO.inspect({item["url"], item["result"]}) "failed" -> IO.inspect({item["url"], item["error"]}) _ -> :ok end end ``` **Item statuses:** `pending` · `running` · `completed` · `failed` **Batch statuses:** `pending` · `running` · `completed` · `failed` · `cancelled` ### batch.submit() + batch.get() ```elixir theme={null} {:ok, %{"batchId" => batch_id}} = Spidra.Batch.submit(config, %{ urls: ["https://example.com/1", "https://example.com/2"], prompt: "Extract the page title" }) # Come back later {:ok, result} = Spidra.Batch.get(config, batch_id) ``` ### Retry failed items Re-queue only the items that failed — successful items are not re-run. ```elixir theme={null} {:ok, result} = Spidra.Batch.get(config, batch_id) if result["failedCount"] > 0 do {:ok, retried} = Spidra.Batch.retry(config, batch_id) IO.puts("Retried #{retried["retriedCount"]} items") end ``` ### Cancel a batch Stops all pending items and refunds credits for unprocessed work. ```elixir theme={null} {:ok, response} = Spidra.Batch.cancel(config, batch_id) IO.puts("Cancelled #{response["cancelledItems"]} items, refunded #{response["creditsRefunded"]} credits") ``` ### List past batches ```elixir theme={null} {:ok, response} = Spidra.Batch.list(config, page: 1, limit: 20) for job <- response["jobs"] do IO.puts("#{job["uuid"]} #{job["status"]} #{job["completedCount"]}/#{job["totalUrls"]}") end ``` *** ## Crawling Give Spidra a starting URL and instructions for which links to follow. It discovers pages automatically, extracts structured data from each one, and returns everything when the crawl is done. ```elixir theme={null} {:ok, job} = Spidra.Crawl.run(config, %{ base_url: "https://competitor.com/blog", crawl_instruction: "Find all blog posts published in 2024", transform_instruction: "Extract the title, author, publish date, and a one-sentence summary", max_pages: 30, use_proxy: true }) for page <- job["result"] do IO.inspect({page["url"], page["data"]}) end ``` **Parameters** | Parameter | Type | Description | | ----------------------- | ------- | -------------------------------------------------- | | `base_url` | string | Starting URL for the crawl | | `crawl_instruction` | string | Which links to follow and which to skip | | `transform_instruction` | string | What to extract from each page | | `max_pages` | integer | Maximum number of pages to crawl | | `use_proxy` | boolean | Route through a residential proxy | | `proxy_country` | string | Two-letter country code, e.g. `"us"` | | `cookies` | string | Raw `Cookie` header string for authenticated sites | ### crawl.submit() + crawl.get() ```elixir theme={null} {:ok, %{"jobId" => job_id}} = Spidra.Crawl.submit(config, %{ base_url: "https://example.com/docs", crawl_instruction: "Follow all documentation pages", transform_instruction: "Extract the page title and a short summary", max_pages: 50 }) # Poll manually {:ok, status} = Spidra.Crawl.get(config, job_id) # status: "waiting" | "active" | "running" | "completed" | "failed" ``` ### Download crawled content Returns signed S3 URLs for the raw HTML and Markdown of each crawled page. Links expire after **1 hour**. ```elixir theme={null} {:ok, response} = Spidra.Crawl.pages(config, job_id) for page <- response["pages"] do IO.puts("#{page["url"]} - #{page["status"]}") # Download raw HTML: page["html_url"] # Download Markdown: page["markdown_url"] end ``` ### Re-extract without re-crawling Apply a new AI prompt to an existing completed crawl without fetching the pages again. Only transformation credits are charged. ```elixir theme={null} {:ok, queued} = Spidra.Crawl.extract(config, source_job_id, "Extract only the product SKUs and prices as a CSV") # Poll the new extraction job {:ok, result} = Spidra.Crawl.get(config, queued["jobId"]) ``` ### History and stats ```elixir theme={null} {:ok, response} = Spidra.Crawl.history(config, page: 1, limit: 10) {:ok, stats} = Spidra.Crawl.stats(config) IO.puts("Total crawls: #{stats["total"]}") ``` *** ## Logs Every API scrape job is logged automatically. Access your full history with optional filters. ```elixir theme={null} {:ok, response} = Spidra.Logs.list(config, %{ status: "failed", search_term: "amazon.com", channel: "api", date_start: "2024-01-01", date_end: "2024-12-31", page: 1, limit: 20 }) for log <- response["logs"] do IO.puts("#{hd(log["urls"])["url"]} #{log["status"]} #{log["credits_used"]}") end ``` **Filter parameters** | Parameter | Type | Description | | ------------- | ------- | --------------------------------------------- | | `status` | string | `"success"` or `"failed"` | | `search_term` | string | Search by URL or prompt | | `channel` | string | `"api"` or `"playground"` | | `date_start` | string | ISO date — return logs on or after this date | | `date_end` | string | ISO date — return logs on or before this date | | `page` | integer | Page number (default: 1) | | `limit` | integer | Results per page (default: 20) | Get a single log entry including the full AI extraction result: ```elixir theme={null} {:ok, log} = Spidra.Logs.get(config, "log-uuid") IO.inspect(log["result_data"]) # the full AI output for that job ``` ## Usage statistics Returns credit and request usage broken down by day or week. ```elixir theme={null} # Range options: "7d" | "30d" | "weekly" {:ok, rows} = Spidra.Usage.get(config, "30d") for row <- rows do IO.puts("#{row["date"]} Requests: #{row["requests"]} Credits: #{row["credits"]}") end ``` | Range | Description | | ---------- | ------------------------------ | | `"7d"` | Last 7 days, one row per day | | `"30d"` | Last 30 days, one row per day | | `"weekly"` | Last 7 weeks, one row per week | Official Python SDK — async-first with sync wrappers. Works in scripts, Django, Flask, and Jupyter. Official .NET SDK — fully async, typed exceptions, JSON schema support. Requires .NET 8+. # Go Source: https://docs.spidra.io/sdks/go Official Go SDK for the Spidra web scraping API. Fully typed structs, zero external dependencies. Scrape, batch, crawl, and run browser actions from Go. The Go SDK wraps the Spidra API with fully typed request and response structs. No `map[string]any` to wrestle with — you get concrete types for everything, errors as values, and zero external dependencies. Just the standard library. ## Installation ```bash theme={null} go get github.com/spidra-io/spidra-go ``` Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Store it as an environment variable — never hardcode it in source. ## Getting started ```go theme={null} import spidra "github.com/spidra-io/spidra-go" client := spidra.New(os.Getenv("SPIDRA_API_KEY")) ``` If you need to override the base URL (for local dev or self-hosted) or set a custom HTTP timeout: ```go theme={null} client := spidra.New( os.Getenv("SPIDRA_API_KEY"), spidra.WithBaseURL("http://localhost:4321/api"), spidra.WithTimeout(30 * time.Second), ) ``` Everything lives under `client.Scrape`, `client.Batch`, `client.Crawl`, `client.Logs`, and `client.Usage`. ## Scraping Each scrape request can include up to three URLs and runs them in parallel. You tell the AI what to extract through a `Prompt`, and optionally lock the output shape with a `Schema`. The quickest path is `Run()` — it submits the job, polls until it completes, and returns the full result: ```go theme={null} job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{ {URL: "https://example.com/pricing"}, }, Prompt: "Extract all pricing plans with name, price, and included features", Output: "json", }) if err != nil { return err } fmt.Println(job.Result.Content) ``` When you need more control — say you're building a queue and want to poll on your own schedule — use `Submit()` and `Get()` separately: ```go theme={null} job, err := client.Scrape.Submit(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{{URL: "https://example.com"}}, Prompt: "Extract the main headline", }) if err != nil { return err } // Come back later and check status, err := client.Scrape.Get(ctx, job.JobID) if err != nil { return err } if status.Status == "completed" { fmt.Println(status.Result.Content) } ``` Jobs move through `waiting` → `active` → `completed` (or `failed`). ### ScrapeParams fields | Field | Type | Description | | -------------------- | ------------- | --------------------------------------------------------------- | | `URLs` | `[]ScrapeURL` | Up to 3 URLs. Each can carry its own `Actions` slice | | `Prompt` | `string` | What to extract, in plain English | | `Output` | `string` | `"markdown"` (default) or `"json"` | | `Schema` | `any` | JSON Schema object — locks the output shape when using `"json"` | | `UseProxy` | `bool` | Route through a residential proxy | | `ProxyCountry` | `string` | Two-letter country code: `"us"`, `"de"`, `"jp"`, etc. | | `ExtractContentOnly` | `bool` | Strip nav, ads, and boilerplate before the AI sees the page | | `Screenshot` | `bool` | Capture a viewport screenshot | | `FullPageScreenshot` | `bool` | Capture a full-page (scrolled) screenshot | | `Cookies` | `string` | Raw `Cookie` header string for pages behind a login | ### Enforcing an exact output shape Pass a `Schema` when you need the output to match a specific structure. Fields the AI can't find come back as `null` instead of guessed values: ```go theme={null} job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{{URL: "https://jobs.example.com/senior-engineer"}}, Prompt: "Extract the job listing details", Output: "json", Schema: map[string]any{ "type": "object", "required": []string{"title", "company", "remote"}, "properties": map[string]any{ "title": map[string]any{"type": "string"}, "company": map[string]any{"type": "string"}, "remote": map[string]any{"type": []string{"boolean", "null"}}, "salary_min": map[string]any{"type": []string{"number", "null"}}, "skills": map[string]any{"type": "array", "items": map[string]any{"type": "string"}}, }, }, }) ``` ### Geo-targeted scraping Some sites return different prices or content based on the visitor's location. Set `UseProxy: true` and pick a `ProxyCountry` to route through a residential IP in that region: ```go theme={null} job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{{URL: "https://www.amazon.de/gp/bestsellers"}}, Prompt: "List the top 10 products with name and price", UseProxy: true, ProxyCountry: "de", }) ``` Supported codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, `sg`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing without pinning to a specific country. ### Scraping pages behind a login Pass your session cookies as a raw header string. The easiest way to grab this is to open browser devtools, log in, and copy the `Cookie` header from any authenticated request: ```go theme={null} job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{{URL: "https://app.example.com/dashboard"}}, Prompt: "Extract the monthly revenue and active user count", Cookies: "session=abc123; auth_token=xyz789", }) ``` ### Browser actions For pages that need interaction before you can extract anything — accepting cookies, typing into a search input, scrolling to trigger lazy loading — include an `Actions` slice on the URL. They run in order before the AI sees the page: ```go theme={null} job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{ { URL: "https://example.com/products", Actions: []map[string]any{ {"type": "click", "selector": "#accept-cookies"}, {"type": "wait", "duration": 1000}, {"type": "scroll", "to": "80%"}, }, }, }, Prompt: "Extract all product names and prices visible on the page", }) ``` For `selector` pass a CSS selector or XPath. If you'd rather describe the element in words, use `value` — Spidra will locate it using AI. | Action | What it does | | --------- | ----------------------------------------------------------------------- | | `click` | Click any element — CSS selector via `selector`, plain text via `value` | | `type` | Type into an input or textarea | | `check` | Check a checkbox | | `uncheck` | Uncheck a checkbox | | `wait` | Pause for `duration` milliseconds | | `scroll` | Scroll to a percentage of page height (e.g. `"80%"`) | | `forEach` | Loop over every matched element and process each one | ### Controlling how long Run() waits By default `Run()` polls every 3 seconds and times out after 120 seconds. Pass `PollOptions` to override: ```go theme={null} job, err := client.Scrape.Run(ctx, params, spidra.PollOptions{ PollInterval: 5 * time.Second, Timeout: 60 * time.Second, }) ``` On timeout, `Run()` returns the result with `Status: "timeout"` rather than an error — the `JobID` is still there so you can keep polling with `Get()` if you need to wait longer. The same options work on `Batch.Run()` and `Crawl.Run()`. ## Batch scraping When you have a list of URLs to process, batch is the right tool. Submit up to 50 URLs in a single request and they run in parallel. Unlike the scraper, each URL here is a plain string — no per-URL actions. ```go theme={null} batch, err := client.Batch.Run(ctx, spidra.BatchParams{ URLs: []string{ "https://shop.example.com/product/1", "https://shop.example.com/product/2", "https://shop.example.com/product/3", }, Prompt: "Extract product name, price, and whether it is in stock", Output: "json", UseProxy: true, }) if err != nil { return err } fmt.Printf("%d/%d completed\n", batch.CompletedCount, batch.TotalURLs) for _, item := range batch.Items { if item.Status == "completed" { fmt.Println(item.URL, item.Result) } else if item.Status == "failed" { fmt.Println("failed:", item.URL, item.Error) } } ``` Each item moves through `pending` → `running` → `completed` (or `failed`). The batch as a whole follows the same lifecycle plus a `cancelled` state. For fire-and-forget, use `Submit()` and come back with `Get()`: ```go theme={null} batch, err := client.Batch.Submit(ctx, spidra.BatchParams{ URLs: []string{"https://example.com/1", "https://example.com/2"}, Prompt: "Extract the page title and meta description", }) if err != nil { return err } // Later... result, err := client.Batch.Get(ctx, batch.BatchID) fmt.Printf("%s: %d/%d done\n", result.Status, result.CompletedCount, result.TotalURLs) ``` ### Retrying failures If some items fail, you can re-queue just those without touching the ones that already succeeded: ```go theme={null} result, _ := client.Batch.Get(ctx, batchID) if result.FailedCount > 0 { if err := client.Batch.Retry(ctx, batchID); err != nil { return err } } ``` To stop a running batch and get credits back for anything that hasn't started: ```go theme={null} if err := client.Batch.Cancel(ctx, batchID); err != nil { return err } ``` To browse past batches: ```go theme={null} page, err := client.Batch.List(ctx, 1, 20) // page, limit for _, job := range page.Jobs { fmt.Printf("%s %s — %d/%d\n", job.UUID, job.Status, job.CompletedCount, job.TotalURLs) } ``` *** ## Crawling Crawling is for when you need to cover a whole site or section, not just a handful of URLs. You give it a starting URL and instructions for which links to follow; it discovers pages on its own, extracts data from each one, and hands everything back when it's done. ```go theme={null} job, err := client.Crawl.Run(ctx, spidra.CrawlParams{ BaseURL: "https://competitor.com/blog", CrawlInstruction: "Follow links to blog posts only — skip tag pages, category pages, and the homepage", TransformInstruction: "Extract the post title, author name, publish date, and a one-sentence summary", MaxPages: 30, UseProxy: true, }) if err != nil { return err } for _, page := range job.Result { fmt.Println(page.URL, page.Data) } ``` `CrawlInstruction` controls navigation — which links to follow, which to skip. `TransformInstruction` controls extraction — what to pull from each page. `MaxPages` is a cap so a crawl doesn't run forever. The same `UseProxy`, `ProxyCountry`, and `Cookies` options from the scraper all work here. For fire-and-forget: ```go theme={null} job, err := client.Crawl.Submit(ctx, spidra.CrawlParams{ BaseURL: "https://example.com/docs", CrawlInstruction: "Follow all documentation pages", TransformInstruction: "Extract the page title and a short summary", MaxPages: 50, }) // Poll later status, err := client.Crawl.Get(ctx, job.JobID) // status.Status: "waiting" | "active" | "running" | "completed" | "failed" ``` ### Downloading the raw content Once a crawl is complete, you can get signed URLs to download the raw HTML and Markdown for every page that was visited. Links expire after an hour: ```go theme={null} result, err := client.Crawl.Pages(ctx, jobID) for _, page := range result.Pages { fmt.Println(page.URL, page.Status) // page.HTMLURL — download the raw HTML // page.MarkdownURL — download the cleaned Markdown } ``` ### Re-extracting with a different prompt Crawling is the expensive part. If you've already crawled a site and just want to pull out different information, you don't have to crawl again — `Extract()` runs a new AI pass over the already-stored content and only charges transformation credits: ```go theme={null} result, err := client.Crawl.Extract( ctx, completedJobID, "Extract product SKUs and prices as structured JSON", ) if err != nil { return err } // This creates a new job — poll it like any other extracted, err := client.Crawl.Get(ctx, result.JobID) ``` ### History and stats ```go theme={null} history, err := client.Crawl.History(ctx, 1, 10) // page, limit fmt.Printf("%d total crawl jobs\n", history.Total) stats, err := client.Crawl.Stats(ctx) fmt.Printf("%d all-time crawls\n", stats.Total) ``` ## Logs Every scrape request your API key makes is logged automatically. You can query the full history and filter by status, URL, date range, or channel: ```go theme={null} result, err := client.Logs.List(ctx, map[string]string{ "status": "failed", "searchTerm": "amazon.com", "dateStart": "2024-01-01", "dateEnd": "2024-12-31", "page": "1", "limit": "20", }) if err != nil { return err } for _, entry := range result.Logs { fmt.Println(entry.URLs[0].URL, entry.Status, entry.CreditsUsed) } ``` To get the full details of a single log — including the AI output from that job: ```go theme={null} entry, err := client.Logs.Get(ctx, logUUID) fmt.Println(entry.ResultData) ``` ## Usage statistics Check how many requests and credits your account has consumed over a period: ```go theme={null} result, err := client.Usage.Get(ctx, "30d") // "7d" | "30d" | "weekly" if err != nil { return err } for _, row := range result.Data { fmt.Printf("%s: %d requests, %d credits\n", row.Date, row.Requests, row.Credits) } ``` `"7d"` gives one row per day for the past week. `"30d"` gives the last 30 days. `"weekly"` gives one row per week for the past seven weeks. ## Error handling All API errors are returned as typed error values. Use `errors.As()` to match the specific type you care about: ```go theme={null} import "errors" _, err := client.Scrape.Run(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{{URL: "https://example.com"}}, Prompt: "Extract the main headline", }) if err != nil { var authErr *spidra.AuthenticationError var credErr *spidra.InsufficientCreditsError var rateErr *spidra.RateLimitError var srvErr *spidra.ServerError var apiErr *spidra.SpidraError switch { case errors.As(err, &authErr): log.Fatal("invalid or missing API key") case errors.As(err, &credErr): log.Fatal("account is out of credits") case errors.As(err, &rateErr): log.Fatal("rate limited — slow down") case errors.As(err, &srvErr): log.Printf("server error — safe to retry: %s", srvErr.Message) case errors.As(err, &apiErr): log.Printf("api error %d: %s", apiErr.StatusCode, apiErr.Message) default: log.Fatal(err) } } ``` | Type | Status | When | | --------------------------- | ------ | ----------------------------------- | | `*AuthenticationError` | 401 | The API key is missing or invalid | | `*InsufficientCreditsError` | 403 | No credits remaining on the account | | `*RateLimitError` | 429 | Too many requests — back off | | `*ServerError` | 500 | Unexpected server-side error | | `*SpidraError` | any | Base type — all others embed this | Every error type exposes `StatusCode int` and `Message string`. `*SpidraError` is the base — if you only want one catch-all, match against that. Official PHP SDK — idiomatic helpers, typed exceptions, and configurable polling. Official Ruby SDK — pure stdlib, no external dependencies. Works in Rails, Sinatra, and scripts. # Java Source: https://docs.spidra.io/sdks/java Official Java SDK for Spidra — AI-powered web scraping with proxy rotation and CAPTCHA handling. The official Java SDK for Spidra lets you extract structured data from any website by describing what you want in plain English. It handles JavaScript rendering, anti-bot bypass, and CAPTCHA solving as a managed API, so your Java code stays focused on the data. * **Java 17+** — uses `java.net.http.HttpClient`, no extra HTTP dependencies * **Jackson** for JSON (de)serialization * **`CompletableFuture`** for all async operations * **Builder pattern** for all request parameter objects ## Installation ### Gradle ```groovy theme={null} dependencies { implementation 'io.spidra:spidra-java-sdk:0.1.0' } ``` ### Maven ```xml theme={null} io.spidra spidra-java-sdk 0.1.0 ``` Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Keep your key out of source control — read it from an environment variable or a secrets manager. *** ## Getting started All requests require an API key. Pass it to the client: ```java theme={null} SpidraClient client = new SpidraClient(System.getenv("SPIDRA_API_KEY")); ``` ## Quick start ```java theme={null} import io.spidra.sdk.SpidraClient; import io.spidra.sdk.model.scrape.ScrapeParams; SpidraClient client = new SpidraClient("your-api-key"); ScrapeParams params = ScrapeParams.builder() .url("https://example.com") .prompt("Extract the page title and main heading") .build(); // submit + poll until complete (non-blocking, returns CompletableFuture) client.scrape().run(params) .thenAccept(job -> System.out.println(job.getResult().getContent())) .exceptionally(err -> { err.printStackTrace(); return null; }) .join(); // block the main thread for this example ``` `run()` submits the job and polls until it completes. The `CompletableFuture` resolves with the final result. *** ## Scraping ### Single-page scrape ```java theme={null} import io.spidra.sdk.SpidraClient; import io.spidra.sdk.model.scrape.ScrapeParams; import io.spidra.sdk.model.scrape.ScrapeJob; SpidraClient client = new SpidraClient(System.getenv("SPIDRA_API_KEY")); ScrapeParams params = ScrapeParams.builder() .url("https://news.ycombinator.com") .prompt("Extract the top 10 story titles and their URLs") .outputFormat("json") .build(); ScrapeJob job = client.scrape().run(params).join(); System.out.println("Status: " + job.getStatus()); System.out.println("Content: " + job.getResult().getContent()); System.out.println("Extracted data: " + job.getResult().getData()); ``` **Job statuses:** `waiting` · `active` · `completed` · `failed` ### Submit and poll manually If you need to track progress yourself, use `submit()` and `get()` directly: ```java theme={null} // Step 1: submit ScrapeJob pending = client.scrape().submit(params).join(); System.out.println("Job submitted: " + pending.getJobId()); // Step 2: poll manually ScrapeJob current; do { Thread.sleep(2000); current = client.scrape().get(pending.getJobId()).join(); System.out.println("Status: " + current.getStatus()); } while (!current.isTerminal()); ``` ### Browser actions ```java theme={null} import io.spidra.sdk.model.scrape.BrowserAction; import java.util.List; ScrapeParams params = ScrapeParams.builder() .url("https://example.com/login") .browserActions(List.of( BrowserAction.builder().type("type").selector("#email").value("user@example.com").build(), BrowserAction.builder().type("type").selector("#password").value("secret").build(), BrowserAction.builder().type("click").selector("button[type=submit]").build(), BrowserAction.builder().type("wait").waitMs(2000).build() )) .prompt("Extract the user dashboard summary") .build(); ScrapeJob job = client.scrape().run(params).join(); ``` **Available actions** | Action | Required fields | Description | | --------- | --------------------- | --------------------------------------------- | | `click` | `selector` or `value` | Click a button, link, or any element | | `type` | `selector`, `value` | Type text into an input or textarea | | `check` | `selector` or `value` | Check a checkbox | | `uncheck` | `selector` or `value` | Uncheck a checkbox | | `wait` | `waitMs` (ms) | Pause for a set number of milliseconds | | `scroll` | `to` (`0–100%`) | Scroll the page to a percentage of its height | *** ## Batch scraping Submit up to 50 URLs in a single request. All URLs are processed in parallel. ```java theme={null} import io.spidra.sdk.model.batch.BatchParams; import io.spidra.sdk.model.batch.BatchJob; import io.spidra.sdk.model.scrape.ScrapeUrl; import java.util.List; BatchParams params = BatchParams.builder() .urls(List.of( ScrapeUrl.builder().url("https://example.com/page1").build(), ScrapeUrl.builder().url("https://example.com/page2").build(), ScrapeUrl.builder().url("https://example.com/page3").build() )) .prompt("Extract the article title, author, and publication date") .outputFormat("json") .build(); BatchJob job = client.batch().run(params).join(); System.out.println("Completed: " + job.getCompletedCount() + "/" + job.getTotal()); job.getItems().forEach(item -> { System.out.println(item.getUrl() + " -> " + item.getStatus()); if ("completed".equals(item.getStatus())) { System.out.println(" Data: " + item.getResult().getData()); } }); ``` **Item statuses:** `pending` · `running` · `completed` · `failed` **Batch statuses:** `pending` · `running` · `completed` · `failed` · `cancelled` ### Cancel a batch ```java theme={null} BatchCancelResult result = client.batch().cancel(batchId).join(); System.out.println("Cancelled: " + result.getCancelledItems() + " items"); System.out.println("Refunded: " + result.getCreditsRefunded() + " credits"); ``` *** ## Crawling Give Spidra a starting URL and instructions for which links to follow. It discovers pages automatically and extracts structured data from each one. ```java theme={null} import io.spidra.sdk.model.crawl.CrawlParams; import io.spidra.sdk.model.crawl.CrawlJob; import io.spidra.sdk.model.crawl.CrawlPagesResult; CrawlParams params = CrawlParams.builder() .url("https://example.com") .maxDepth(3) .maxPages(100) .includePatterns(List.of("/blog/*")) .excludePatterns(List.of("/tag/*", "/author/*")) .prompt("Extract the blog post title and summary") .build(); CrawlJob job = client.crawl().run(params).join(); System.out.println("Crawled " + job.getPagesCrawled() + " pages"); ``` ### Retrieve all crawled pages ```java theme={null} CrawlPagesResult pages = client.crawl().pages(job.getJobId()).join(); pages.getPages().forEach(page -> { System.out.println(page.getUrl() + " [depth=" + page.getDepth() + "]"); if (page.getData() != null) { System.out.println(" Extracted: " + page.getData()); } }); ``` ### Re-extract without re-crawling Apply a new AI prompt to an existing completed crawl without fetching pages again. ```java theme={null} Object extracted = client.crawl() .extract(jobId, "Summarize all blog posts into a single markdown document") .join(); System.out.println(extracted); ``` *** ## Logs Every API scrape job is logged automatically. ```java theme={null} import io.spidra.sdk.model.logs.LogsParams; import io.spidra.sdk.model.logs.LogsResult; import io.spidra.sdk.model.logs.ScrapeLogDetail; LogsParams params = LogsParams.builder() .status("completed") .channel("production") .limit(25) .page(1) .dateStart("2024-01-01T00:00:00Z") .build(); LogsResult result = client.logs().list(params).join(); System.out.println("Total logs: " + result.getPagination().get("total")); // Get full detail for a single log ScrapeLogDetail detail = client.logs().get(result.getLogs().get(0).getUuid()).join(); System.out.println("Prompt used: " + detail.getPrompt()); ``` *** ## Usage statistics Returns credit and request usage broken down by day or week. ```java theme={null} import io.spidra.sdk.model.usage.UsageStats; UsageStats stats = client.usage().get("30d").join(); System.out.println("Plan: " + stats.getPlan()); System.out.println("Credits used (30d): " + stats.getCreditsUsed()); System.out.println("Credits remaining: " + stats.getCreditsRemaining()); ``` | Range | Description | | ---------- | ------------------------------ | | `"7d"` | Last 7 days, one row per day | | `"30d"` | Last 30 days, one row per day | | `"weekly"` | Last 7 weeks, one row per week | ## Error handling All exceptions extend `SpidraException` (an unchecked `RuntimeException`): | Exception | HTTP Status | | ------------------------------------ | ----------- | | `SpidraAuthException` | 401, 403 | | `SpidraInsufficientCreditsException` | 402 | | `SpidraRateLimitException` | 429 | | `SpidraServerException` | 5xx | | `SpidraException` | any other | ```java theme={null} import io.spidra.sdk.exception.*; client.scrape().run(params) .thenAccept(job -> System.out.println(job.getResult().getContent())) .exceptionally(throwable -> { Throwable cause = throwable.getCause(); if (cause instanceof SpidraAuthException) { System.err.println("Invalid API key"); } else if (cause instanceof SpidraRateLimitException) { System.err.println("Rate limited — back off and retry"); } else if (cause instanceof SpidraInsufficientCreditsException) { System.err.println("Out of credits"); } else if (cause instanceof SpidraException ex) { System.err.println("API error [" + ex.getStatusCode() + "]: " + ex.getMessage()); } return null; }); ``` *** ## Configuration ### Custom base URL ```java theme={null} // Point at a staging environment or local mock SpidraClient client = new SpidraClient("your-api-key", "https://staging-api.spidra.io/api"); ``` ### Environment variable pattern ```java theme={null} SpidraClient client = new SpidraClient( Objects.requireNonNull(System.getenv("SPIDRA_API_KEY"), "SPIDRA_API_KEY env var not set") ); ``` ## Building from source ```bash theme={null} ./gradlew build ./gradlew test ./gradlew javadoc ``` Official Swift SDK — async/await native, works on iOS, macOS, tvOS, watchOS, and server-side Swift. Official Rust SDK — tokio-based async, zero-cost abstractions, returns Result on every call. # LangChain Source: https://docs.spidra.io/sdks/langchain Use Spidra as a document loader or agent tool inside LangChain pipelines. `langchain-spidra` integrates Spidra into LangChain in two ways: * **Document loader.** `SpidraLoader` pulls web content as LangChain `Document` objects, ready to feed into a vector store or retrieval chain. * **Agent tools.** `SpidraScrape`, `SpidraCrawl`, and `SpidraBatchScrape` are `BaseTool` classes you can bind to any LangChain agent so it can browse the web on its own. ## Installation ```bash theme={null} pip install langchain-spidra ``` Requires Python 3.9 or higher. This package is for LangChain's Python framework; if you are building agents in JavaScript, see the [Node SDK's AI agent integration](/sdks/node#ai-agent-integration) instead. Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**. Set it as an environment variable so you never hardcode it. ```bash theme={null} export SPIDRA_API_KEY="spd_your-api-key" ``` ## Document loader `SpidraLoader` implements LangChain's `BaseLoader` interface and returns a list of `Document` objects, each with `page_content` and `metadata`. ### Scrape a page The default mode. Returns raw markdown, which is ideal for RAG indexing where you want clean, unmodified content: ```python theme={null} from langchain_spidra import SpidraLoader loader = SpidraLoader(url="https://example.com") docs = loader.load() print(docs[0].page_content[:200]) print(docs[0].metadata) # {"source": "https://example.com"} ``` Pass up to three URLs at once and get one document per URL: ```python theme={null} loader = SpidraLoader( urls=["https://example.com/pricing", "https://example.com/features"], ) docs = loader.load() # len(docs) == 2 ``` #### AI extraction Add a `prompt` to run AI extraction. You get back a single document containing the structured result instead of raw markdown: ```python theme={null} loader = SpidraLoader( url="https://example.com/pricing", params={ "prompt": "Extract all pricing plans: name, price, and included features", "output": "json", }, ) docs = loader.load() # docs[0].page_content == '{"plans": [...]}' # docs[0].metadata["ai_extraction_failed"] == False ``` ### Crawl a site Set `mode="crawl"` to crawl an entire site. Returns one document per discovered page, which is exactly what you want for indexing a docs site or knowledge base: ```python theme={null} loader = SpidraLoader( url="https://docs.example.com", mode="crawl", params={ "crawl_instruction": "Follow all documentation pages", "max_pages": 50, }, ) docs = loader.load() for doc in docs: print(doc.metadata["source"], doc.page_content[:100]) ``` Omitting `transform_instruction` returns raw markdown per page with no AI processing and no token charges. Pass it only when you need AI extraction on each crawled page. With AI extraction per page: ```python theme={null} loader = SpidraLoader( url="https://competitor.com/blog", mode="crawl", params={ "crawl_instruction": "Follow links to blog posts only", "transform_instruction": "Extract the post title, author, and a one-sentence summary", "max_pages": 30, }, ) docs = loader.load() ``` ### Batch scrape Set `mode="batch"` to scrape many URLs in parallel. Returns one document per completed URL: ```python theme={null} loader = SpidraLoader( urls=[ "https://shop.example.com/product/1", "https://shop.example.com/product/2", "https://shop.example.com/product/3", ], mode="batch", ) docs = loader.load() ``` ### Use with a vector store The most common use case: crawl a site and build a searchable knowledge base. ```python theme={null} from langchain_spidra import SpidraLoader from langchain_openai import OpenAIEmbeddings from langchain_community.vectorstores import FAISS loader = SpidraLoader( url="https://docs.example.com", mode="crawl", params={"crawl_instruction": "Crawl all documentation pages", "max_pages": 50}, ) docs = loader.load() vectorstore = FAISS.from_documents(docs, OpenAIEmbeddings()) retriever = vectorstore.as_retriever() ``` ### Lazy loading For large crawls, use `lazy_load()` to process documents as they arrive instead of waiting for everything to finish: ```python theme={null} loader = SpidraLoader(url="https://docs.example.com", mode="crawl") for doc in loader.lazy_load(): # process one document at a time print(doc.metadata["source"]) ``` ### Loader parameters | Parameter | Type | Default | Description | | --------- | ---------- | ---------- | ------------------------------------------------------------------------- | | `url` | str | none | URL to scrape or crawl (required for `scrape` and `crawl` modes) | | `urls` | list\[str] | none | List of URLs (required for `batch` mode; optional multi-URL for `scrape`) | | `api_key` | str | env var | Spidra API key. Falls back to `SPIDRA_API_KEY` | | `api_url` | str | none | Override the API base URL (self-hosted instances) | | `mode` | str | `"scrape"` | One of `"scrape"`, `"crawl"`, or `"batch"` | | `params` | dict | `{}` | Extra parameters forwarded to the underlying SDK method | *** ## Agent tools Each Spidra capability is available as a standalone `BaseTool` you can bind to an agent. ### Available tools | Tool | What it does | | ------------------- | --------------------------------------------------------------- | | `SpidraScrape` | Scrapes a single URL and returns its markdown content | | `SpidraCrawl` | Crawls a site from a starting URL, returns all discovered pages | | `SpidraBatchScrape` | Scrapes a list of URLs in parallel, returns one result per URL | ### Standalone usage ```python theme={null} from langchain_spidra import SpidraScrape, SpidraCrawl, SpidraBatchScrape # Scrape a page scrape = SpidraScrape() result = scrape.invoke({"url": "https://example.com"}) print(result["content"]) # Crawl a site crawl = SpidraCrawl() pages = crawl.invoke({"url": "https://docs.example.com", "max_pages": 20}) for page in pages: print(page["url"], page["data"][:100]) # Batch scrape batch = SpidraBatchScrape() items = batch.invoke({"urls": ["https://a.com", "https://b.com"]}) for item in items: print(item["url"], item["status"], item["result"][:100]) ``` **What each tool returns:** | Tool | Returns | | ------------------- | --------------------------------------------------------------------------------------- | | `SpidraScrape` | A dict: `content` (markdown or JSON string), `ai_extraction_failed`, `extraction_empty` | | `SpidraCrawl` | A list of page dicts: `url`, `data`, `title` | | `SpidraBatchScrape` | A list of item dicts: `url`, `result`, `status` (`"completed"` or `"failed"`) | **Error handling:** the tools never raise. On failure they return the error as a plain string, so the agent can read what went wrong and adjust. When calling a tool yourself, check the type before indexing into the result: ```python theme={null} result = scrape.invoke({"url": "https://example.com"}) if isinstance(result, str): print("Scrape failed:", result) else: print(result["content"]) ``` ### Bind to an agent ```python theme={null} from langchain_anthropic import ChatAnthropic from langchain.agents import AgentExecutor, create_tool_calling_agent from langchain_core.prompts import ChatPromptTemplate from langchain_spidra import SpidraScrape, SpidraCrawl tools = [SpidraScrape(), SpidraCrawl()] llm = ChatAnthropic(model="claude-opus-4-8") prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful research assistant with access to web scraping tools."), ("human", "{input}"), ("placeholder", "{agent_scratchpad}"), ]) agent = create_tool_calling_agent(llm, tools, prompt) executor = AgentExecutor(agent=agent, tools=tools, verbose=True) result = executor.invoke({ "input": "What are the main features listed on https://spidra.io?" }) print(result["output"]) ``` ### Tool parameters **`SpidraScrape`** | Parameter | Type | Description | | ---------------------- | ---- | ------------------------------------------------- | | `url` | str | The URL to scrape | | `prompt` | str | AI extraction instruction (enables AI mode) | | `output` | str | Output format: `"markdown"` (default) or `"json"` | | `use_proxy` | bool | Route through a residential proxy | | `proxy_country` | str | Two-letter country code for the proxy | | `extract_content_only` | bool | Strip nav and ads, return main content only | **`SpidraCrawl`** | Parameter | Type | Description | | ----------------------- | ---------- | ------------------------------------------------ | | `url` | str | Starting URL for the crawl | | `crawl_instruction` | str | Which pages to follow, in plain language | | `transform_instruction` | str | What to extract from each page (enables AI mode) | | `max_pages` | int | Maximum number of pages to crawl | | `max_depth` | int | Maximum link depth from the starting URL | | `include_paths` | list\[str] | URL path patterns to include | | `exclude_paths` | list\[str] | URL path patterns to skip | | `use_proxy` | bool | Route through a residential proxy | | `proxy_country` | str | Two-letter country code for the proxy | **`SpidraBatchScrape`** | Parameter | Type | Description | | ---------------------- | ---------- | ------------------------------------------------- | | `urls` | list\[str] | List of URLs to scrape | | `prompt` | str | AI extraction instruction applied to all pages | | `output` | str | Output format: `"markdown"` (default) or `"json"` | | `use_proxy` | bool | Route through a residential proxy | | `proxy_country` | str | Two-letter country code for the proxy | | `extract_content_only` | bool | Strip nav and ads, return main content only | ## Source The package is open-source at [github.com/spidra-io/spidra-langchain](https://github.com/spidra-io/spidra-langchain). Bug reports and contributions welcome. Use the Spidra Python SDK directly without LangChain. Official Node.js SDK for JavaScript and TypeScript projects. # MCP Server Source: https://docs.spidra.io/sdks/mcp Connect Spidra to Claude, Cursor, and any MCP-compatible AI agent. MCP is the standard that lets AI assistants use external tools. The [Spidra MCP server](https://github.com/spidra-io/spidra-mcp-server) gives any MCP-compatible assistant like Claude Code, Claude Desktop, Cursor, Windsurf, and VS Code the ability to scrape pages, batch-process URLs, and crawl entire sites on its own. You describe what you want in chat and the assistant picks the right Spidra tool, runs it, and works with the extracted data directly. You never call the tools by name, and you do not write any code. The server ships with 12 tools. Their descriptions teach the model when to use which, including how to keep your credit spend low. ## Before you start You need three things: 1. **A Spidra API key.** Create one at [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**. Keys start with `spd_`. 2. **Node.js 20 or newer.** Check with `node --version`. The `npx` command that runs the server ships with Node. 3. **An MCP-compatible client.** Any of the assistants below works. ## Setup Every setup below does the same thing. It tells your assistant to run `npx -y spidra-mcp` and hands the server your API key through an environment variable. Run this one command in your terminal, replacing the placeholder with your real key: ```bash theme={null} claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcp ``` Start a new Claude Code session, then run `/mcp` to confirm the connection shows as active. 1. Open Cursor Settings 2. Go to **Features** > **MCP Servers** 3. Click **+ Add new global MCP server** 4. Paste the following and replace the placeholder key: ```json theme={null} { "mcpServers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" } } } } ``` You can also put this in a `.cursor/mcp.json` file inside a project if you only want Spidra available there. 1. Open **Settings** > **Developer** > **Edit Config**. This opens `claude_desktop_config.json` 2. Add the `spidra` entry inside `mcpServers`: ```json theme={null} { "mcpServers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" } } } } ``` 3. Quit and reopen Claude Desktop. The tools appear under the tools icon in the chat input. Press `Ctrl + Shift + P` (or `Cmd + Shift + P` on Mac), type `Preferences: Open User Settings (JSON)`, and add: ```json theme={null} { "mcp": { "servers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" } } } } } ``` To share the setup with your team instead, put the same `servers` block in a `.vscode/mcp.json` file in your repository and use a `promptString` input for the key so it never gets committed. Add to `~/.codeium/windsurf/mcp_config.json`: ```json theme={null} { "mcpServers": { "spidra": { "command": "npx", "args": ["-y", "spidra-mcp"], "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" } } } } ``` ## Try it Once connected, just ask for web data in normal language: > "Scrape [https://news.ycombinator.com](https://news.ycombinator.com) and give me the top 5 stories with their points." > "Compare the pricing pages of stripe.com and paddle.com and tell me which is cheaper for a small SaaS." > "Here are 12 product URLs. Get me the name, price, and rating for each one as a table." > "Crawl the first 10 pages of docs.example.com and summarize what the product does." If the assistant answers with real data from those pages, everything is working. The assistant chooses the tool, waits or polls as needed, and answers with the extracted data. ## How to choose a tool This guidance is embedded in the tool descriptions, so the assistant follows it on its own. It is worth knowing yourself so you can phrase requests well. The deciding question is not how many URLs you have. It is what you want back: * **You want one answer, and you know the URL (or 2 to 3 related URLs):** the assistant uses **`spidra_scrape`**. With several URLs, their content is merged and the AI answers once across all of them. That makes it the right tool for comparing two pricing pages or summarizing three related articles into one answer. * **You want separate data for each URL in a list:** the assistant uses **`spidra_batch_scrape`**, even for just 2 URLs. Every URL is processed independently and returns its own result. * **You do not know the page URLs yet:** the assistant uses **`spidra_crawl`**. It takes one starting URL and a plain-English instruction about which links to follow, and discovers the pages itself. | Tool | Best for | Waits or polls? | | ---------------------------- | ---------------------------------------------------------- | ------------------------------------ | | `spidra_scrape` | One combined answer from 1 to 3 known URLs | Waits, returns the result directly | | `spidra_check_scrape_status` | Re-checking a scrape that outlived its wait window | Instant lookup | | `spidra_batch_scrape` | Separate results for each of 2 to 50 known URLs | Returns a `batchId`, assistant polls | | `spidra_check_batch_status` | Progress and per-URL results for a batch | Instant lookup | | `spidra_cancel_batch` | Stopping a batch you no longer need | Instant | | `spidra_crawl` | Discovering and processing pages from one starting URL | Returns a `jobId`, assistant polls | | `spidra_check_crawl_status` | Progress, then full results, for a crawl | Instant lookup | | `spidra_crawl_pages` | Per-page results with raw HTML and markdown download links | Instant lookup | | `spidra_crawl_extract` | Asking a new question of an already-completed crawl | Returns a new `jobId` | | `spidra_cancel_crawl` | Stopping a crawl you no longer need | Instant | | `spidra_scrape_logs` | Looking up past jobs and their outputs | Instant lookup | | `spidra_usage` | Checking credit and request usage | Instant lookup | When you need specific fields from a page, ask for JSON and describe the fields. The assistant gets back a small, focused payload instead of an entire page. Ask for full markdown only when you genuinely need the whole page. ## The tools in detail Scrapes 1 to 3 URLs and extracts their content with AI. Waits for the result, typically 10 to 60 seconds, and returns the extracted content directly. The important behavior to understand: with more than one URL, the pages' content is combined and the AI produces **one answer across all of them**. The raw per-page content still comes back in `pages`, but the extraction is a single merged result. Use multiple URLs to compare or synthesize across pages. For separate results per URL, the assistant should use `spidra_batch_scrape` instead, even for 2 URLs. **Prompt example:** > "Get the product name, price, and description from [https://example.com/product](https://example.com/product)." **Usage example (structured extraction with a schema):** ```json theme={null} { "name": "spidra_scrape", "arguments": { "urls": ["https://example.com/product"], "prompt": "Extract the product information", "output": "json", "schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "description": { "type": "string" } }, "required": ["name", "price"] } } } ``` **Other options:** `actions` (browser steps to run first: click, type, scroll, forEach loops), `cookies` (for pages behind a login), `useProxy` and `proxyCountry` (residential proxy routing), `screenshot`, `extractContentOnly` (strip boilerplate), and `scrapeMode: "fast"` (HTTP only, no browser, cheaper but cannot run actions). **Returns:** the extracted `content`, per-page raw data in `pages`, any `screenshots`, and `stats` with token counts. If the wait window is exceeded, the job keeps running server-side and the error hands the assistant the job ID to check with `spidra_check_scrape_status`. Submits 2 to 50 URLs processed in parallel with the same prompt or schema. Each URL is handled **independently and gets its own result**. Returns immediately with a `batchId`; the assistant polls `spidra_check_batch_status` every 10 to 15 seconds until the batch finishes. The tool response tells it to do exactly that. **Prompt example:** > "Here are 15 product URLs. Extract the name, price, and star rating from each one." **Usage example:** ```json theme={null} { "name": "spidra_batch_scrape", "arguments": { "urls": [ "https://shop.example.com/product/1", "https://shop.example.com/product/2" ], "prompt": "Extract the product name, price, and star rating", "output": "json" } } ``` **Returns:** `{ "batchId": "...", "total": 2 }` plus polling instructions. URLs are plain strings, not objects. A `completed` batch can still contain individual failed items; check `failedCount` in the status. Failed items can be retried through the [batch retry endpoint](/api-reference/scraping/batch-scrape-retry) without re-running the whole batch. Crawls a website starting from one URL, following links according to a plain-English instruction, and optionally extracting structured data from every page. Returns immediately with a `jobId`; the assistant polls `spidra_check_crawl_status`. Two instructions control a crawl, and they do different jobs: * `crawlInstruction` controls **which links get followed**: "Follow blog post links only, skip tag and category pages." * `transformInstruction` controls **what gets extracted from each page**: "Extract the title, author, and publish date." Leave it out (with no schema) and each page comes back as raw markdown with no AI token cost at all. **Prompt example:** > "Crawl example.com/blog, follow only the article links, and get me each post's title, author, and date. Cap it at 10 pages." **Usage example:** ```json theme={null} { "name": "spidra_crawl", "arguments": { "baseUrl": "https://example.com/blog", "crawlInstruction": "Follow blog post links only, skip tag and category pages", "transformInstruction": "Extract the title, author, and publish date", "maxPages": 10 } } ``` **Scoping options:** `maxPages` (default 5, maximum 50), `maxDepth`, `includePaths` and `excludePaths`, `allowSubdomains`, `crawlEntireDomain`, and `ignoreQueryParams`. Every crawled page costs credits, so start small. You can always crawl again, or better, re-extract. **Returns:** `{ "jobId": "..." }` plus polling instructions. Runs a brand new extraction instruction over a crawl that already completed, without fetching any pages again. Spidra kept the page content, so only AI token credits are charged. This is the cheap way to ask a second question of the same site. For example: you crawled a competitor's blog extracting titles and dates. Now you want the key topics of each post too. Re-extract instead of re-crawling. ```json theme={null} { "name": "spidra_crawl_extract", "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000", "transformInstruction": "List the main topics each post covers" } } ``` **Returns:** a new `jobId`, polled like a normal crawl. The source crawl must have status `completed`. **`spidra_check_scrape_status`** looks up a scrape by job ID. Only needed when a scrape outlived its wait window; the timeout error includes the ID. **`spidra_check_batch_status`** returns batch progress (`completedCount`, `failedCount`) and per-URL results for every finished item. **`spidra_check_crawl_status`** returns crawl progress (`pagesCrawled` out of `maxPages`) while running, and every page's extracted data once completed. **`spidra_crawl_pages`** returns per-page results with signed download URLs for each page's raw HTML and markdown. The links expire after 1 hour. Works on cancelled crawls too, returning whatever finished before the cancellation. **`spidra_cancel_batch`** and **`spidra_cancel_crawl`** stop a running job. Finished work is kept, and credits for the unprocessed portion are refunded automatically. **`spidra_scrape_logs`** browses past jobs: what ran, whether it succeeded, and how many credits it used. Filters by status and search term; pass a `uuid` for one job's full AI output. Ask the assistant "why did my last scrape fail?" and this is what it reaches for. **`spidra_usage`** reports request, credit, and token usage by day or week (`"7d"`, `"30d"`, or `"weekly"`). Useful before kicking off a large batch or crawl. ## How it protects your credits Agent loops can burn credits fast if the tools let them. The server is built to prevent that: * The tool descriptions steer the model toward the cheapest tool that answers the question, and tell it to keep `maxPages` small. * Long-running jobs return a job ID with explicit polling instructions, so the model never resubmits a job that is still running. Timeout errors say the job is still running and to poll, not retry. * Error messages carry retry guidance: rate limits say exactly how long to wait, validation errors list exactly what to fix, and permanent errors say not to retry at all. No expensive retry loops on requests that can never succeed. * Cancelling unfinished work refunds unprocessed items, and the cancel tools say so. ## Output size Large pages and big crawls can produce more text than fits in a model's context window. The server truncates individual strings above 5,000 characters and caps any single tool response at 80,000 characters, marking every truncation clearly. When the assistant needs the complete raw content of crawled pages, `spidra_crawl_pages` provides download links to the full files. ## Configuration | Env var | Required | Description | | ------------------------ | -------- | ---------------------------------------------------------------------------------------------------- | | `SPIDRA_API_KEY` | Yes | Your Spidra API key, starting with `spd_` | | `SPIDRA_API_URL` | No | Override the API base URL, for staging or self-hosted setups | | `HTTP_STREAMABLE_SERVER` | No | Set to `true` to serve the HTTP streamable transport at `http://localhost:3000/mcp` instead of stdio | | `PORT` / `HOST` | No | Bind address for the HTTP transport. Defaults are `3000` and `localhost` | On the HTTP transport, the API key can also be sent per request via an `X-Spidra-API-Key` or `Authorization: Bearer` header, which is useful when one server instance serves more than one user. ## Troubleshooting **The assistant does not see any Spidra tools.** Restart your client after adding the config. Most clients only read MCP configuration at startup. In Claude Code, run `/mcp` to check the connection status. **"No Spidra API key configured."** The `SPIDRA_API_KEY` variable is not reaching the server. Make sure it is inside the `env` block of the server entry, not at the top level of the config file, and that the key still exists under **Settings** > **API Keys** in your dashboard. **A scrape "timed out."** The job is still running server-side and nothing is lost. The error includes the job ID, and the assistant will fetch the result with `spidra_check_scrape_status`. Bot-protected sites can take a couple of minutes. **Results come back empty when using a schema.** Check the schema: every field you want must be defined with a type. An object with no properties gives the AI nothing to fill in. **`npx` cannot find the package.** Make sure you are on Node 20 or newer and that your network allows access to the npm registry. Use Spidra as a document loader or agent tool inside LangChain pipelines. Prefer code over chat? Browse the official SDKs. # n8n Source: https://docs.spidra.io/sdks/n8n Trigger Spidra web scraping and crawling jobs directly from n8n workflows. No code required — connect extracted data to any downstream node. The Spidra n8n node lets you trigger scrape jobs, batch process URLs, and crawl entire websites as steps in any n8n workflow. No code required. Configure your extraction prompt, connect the output to whatever comes next, and you're done. ## Installation In your n8n instance, go to **Settings > Community Nodes** and install: ``` n8n-nodes-spidra ``` Requires n8n 1.0 or higher. After installing, restart n8n for the node to appear in the editor. ## Authentication Add a new **Spidra API** credential and enter your API key. You can get your key from [app.spidra.io](https://app.spidra.io) under **Settings > API Keys**. If you are running a self-hosted Spidra instance, change the **Base URL** field to point at your server. The default is `https://api.spidra.io/api`. ## Resources and operations The node has five resources. Each one maps directly to the Spidra API. | Resource | Operations | | ---------------- | ---------------------------------------------------- | | **Scrape** | Run, Submit, Get Status | | **Batch Scrape** | Run, Submit, Get Status, List, Cancel, Retry Failed | | **Crawl** | Run, Submit, Get Status, Get Pages, Extract, History | | **Logs** | List, Get | | **Usage** | Get Stats | ## Run vs Submit + Get Status Every resource that creates a job has two ways to handle it. **Run** submits the job and keeps the workflow waiting until results come back. This is the simplest option and works well for short jobs. You set a **Max Wait Time** (default 120 seconds). If the job finishes in time, the node outputs the full result. If it times out, the node outputs a `{ status: "timeout", jobId: "..." }` response so you can chain a **Get Status** node and check progress later. **Submit** returns the job ID immediately without waiting. Use this when you want to kick off a long job and check it in a later step or a separate workflow run. ## Scraping Select **Resource: Scrape** and **Operation: Run** to scrape up to three URLs in one request. Add your URLs using the **Add URL** button. Each URL can include an optional **Browser Actions** JSON array for interactions like clicking, scrolling, or filling a form before the AI extracts. Set **Output Format** to JSON or Markdown. Use the **Options** collection to add an extraction prompt, a JSON schema for structured output, proxy settings, cookies, and screenshot capture. **Extraction Prompt** tells the AI what to pull from the page in plain English. **Extraction Schema** enforces an exact output shape and takes precedence over the prompt when both are set. ## Batch scraping Select **Resource: Batch Scrape** to process a large list of URLs in one job. Add each URL as a separate line in the **URLs** field. The batch supports up to 50 URLs per job and processes them all in parallel. The same options available in Scrape (prompt, schema, proxy, cookies, screenshots) are available here too. If some items fail, use **Retry Failed** with the batch ID to re-queue only the failed URLs without re-running the ones that already completed. Use **Cancel** to stop a running batch and get credits refunded for anything that has not started yet. ## Crawling Select **Resource: Crawl** to start from a URL and let Spidra discover and process pages on its own. Three fields are required: * **Start URL**: the root page the crawler starts from * **Navigation Instruction**: plain English instructions for which links to follow and which to skip * **Extraction Instruction**: what data to pull from each page the crawler visits Under **Options**, set **Max Pages** to cap how many pages the crawl visits. Proxy and cookie options work the same as in Scrape. Once a crawl completes, use **Get Pages** with the job ID to retrieve signed download URLs for the raw HTML and Markdown of every crawled page. URLs expire after one hour. Use **Extract** to re-run AI extraction on a completed crawl with a new instruction, without re-crawling any pages. This only charges transformation credits. ## Logs and Usage **Logs: List** returns paginated scrape logs for your account. Filter by status (success, error, in progress) and search by URL or preset name using the **Filters** collection. **Logs: Get** returns the full detail of a single log entry including the AI extraction output. **Usage: Get Stats** returns credit usage, request counts, and bandwidth broken down by day or week. Choose the time window from the **Time Range** dropdown: Last 7 Days, Last 30 Days, or This Week. ## Using as an AI tool The Spidra node has `usableAsTool` enabled, which means you can connect it directly to an **AI Agent** node in n8n. The agent can call Spidra to fetch live web data as part of its reasoning without any additional setup on your end. ## Error handling Enable **Continue On Fail** on the node to prevent a single failed item from stopping the whole workflow. When enabled, errors are returned as `{ error: "..." }` in the output and execution continues with the next item. # Node Source: https://docs.spidra.io/sdks/node Scrape pages, run browser actions, batch-process URLs, and crawl entire sites using the Spidra Node SDK. The SDK handles the plumbing you'd otherwise write yourself like job submission and polling, automatic retries with backoff for transient failures, typed errors for everything that can go wrong, and streaming results for long-running jobs. It has zero dependencies and runs anywhere `fetch` exists like Node 18+, browsers, and edge runtimes like Cloudflare Workers. ## Installation To install the [Spidra Node SDK](https://www.npmjs.com/package/spidra), you can use npm: ```bash theme={null} npm install spidra ``` Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**. Never hardcode it in source files. Use an environment variable instead. ## Setup Here’s an example of initializing the Spidra client in a Node.js or TypeScript project. ```typescript theme={null} import { SpidraClient } from 'spidra'; const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY }); ``` ## Scraping All scrape jobs run asynchronously. `run()` submits a job and polls until it finishes. For manual control, use `submit()` and `get()` directly. Up to **3 URLs** can be passed per request and are processed in parallel. ### Scrape a web page Submit a scrape job and wait for results. ```typescript theme={null} const job = await spidra.scrape.run({ urls: [{ url: 'https://example.com/pricing' }], prompt: 'Extract all pricing plans with name, price, and included features', output: 'json', }); console.log(job.result.content); // { plans: [{ name: "Starter", price: "$9/mo", features: [...] }, ...] } ``` **Parameters** | Parameter | Type | Description | | -------------------- | --------------------------------------- | ------------------------------------------------------------------- | | `urls` | `{ url: string, actions?: Action[] }[]` | URLs to scrape, with optional per-URL browser actions | | `prompt` | `string` | AI extraction instruction | | `output` | `"markdown"` \| `"json"` | Response format. Defaults to `"markdown"` | | `schema` | `object` | JSON Schema for guaranteed output shape (use with `output: "json"`) | | `useProxy` | `boolean` | Route through a residential proxy | | `proxyCountry` | `string` | Two-letter country code, e.g. `"us"`, `"de"`, `"jp"` | | `extractContentOnly` | `boolean` | Strip navigation, ads, and boilerplate before AI extraction | | `screenshot` | `boolean` | Capture a screenshot of the page | | `fullPageScreenshot` | `boolean` | Capture a full-page (scrolled) screenshot | | `cookies` | `string` | Raw `Cookie` header string for authenticated pages | ### Fire-and-forget approach Fire-and-forget approach: submit a job immediately and poll on your own schedule. ```typescript theme={null} // Submit — returns immediately with a jobId const { jobId } = await spidra.scrape.submit({ urls: [{ url: 'https://example.com' }], prompt: 'Extract the main headline', }); // Check status at any time const status = await spidra.scrape.get(jobId); if (status.status === 'completed') { console.log(status.result.content); } else if (status.status === 'failed') { console.error(status.error); } ``` **Job statuses:** `queued` · `waiting` · `active` · `completed` · `failed` ### Structured JSON output Pass a `schema` to enforce an exact output shape. Missing fields come back as `null` rather than hallucinated values — which matters when the output feeds a database or a typed pipeline downstream. Define every field you want extracted. An untyped `object` with no `properties` (or an array of them) gives the AI nothing to fill in, so those members come back empty. Use the [Spidra JSON Schema Generator](https://spidra.io/tools/json-schema-generator) to build and preview your schema visually before pasting it here. ```typescript theme={null} const job = await spidra.scrape.run({ urls: [{ url: 'https://jobs.example.com/senior-engineer' }], prompt: 'Extract the job listing details', output: 'json', schema: { type: 'object', required: ['title', 'company', 'remote'], properties: { title: { type: 'string' }, company: { type: 'string' }, remote: { type: ['boolean', 'null'] }, salary_min: { type: ['number', 'null'] }, salary_max: { type: ['number', 'null'] }, skills: { type: 'array', items: { type: 'string' } }, }, }, }); ``` ### Structured output with Zod If you already use [Zod](https://zod.dev), skip the JSON Schema entirely — pass your Zod schema and the SDK converts it for you. The result is typed from your schema, so `content` needs no casting: ```typescript theme={null} import { z } from 'zod'; const JobListing = z.object({ title: z.string(), company: z.string(), remote: z.boolean().nullable(), skills: z.array(z.string()), }); const job = await spidra.scrape.run({ urls: [{ url: 'https://jobs.example.com/senior-engineer' }], prompt: 'Extract the job listing details', output: 'json', schema: JobListing, }); job.result.content.title; // typed as string — no casting needed ``` The same works on `batch.run()` (types each item's `result`) and `crawl.run()` (types each page's `data`). Zod is an optional peer dependency — install it only if you use this. Passing `MySchema.shape` by mistake throws a helpful error instead of failing silently. ### Geo-targeted scraping Route through a residential proxy in a specific country for geo-restricted content or localized pricing. ```typescript theme={null} const job = await spidra.scrape.run({ urls: [{ url: 'https://www.amazon.de/gp/bestsellers' }], prompt: 'List the top 10 products with name and price', useProxy: true, proxyCountry: 'de', }); ``` Supported codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, `sg`, `es`, `it`, `mx`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing. ### Authenticated pages Pass session cookies as a raw header string to scrape pages behind a login. ```typescript theme={null} const job = await spidra.scrape.run({ urls: [{ url: 'https://app.example.com/dashboard' }], prompt: 'Extract the monthly revenue and active user count', cookies: 'session=abc123; auth_token=xyz789', }); ``` ### Browser actions Run actions against the page before extraction. They execute in order — the scrape happens after all actions complete. ```typescript theme={null} const job = await spidra.scrape.run({ urls: [ { url: 'https://example.com/products', actions: [ { type: 'click', selector: '#accept-cookies' }, { type: 'wait', duration: 1000 }, { type: 'scroll', to: '80%' }, ], }, ], prompt: 'Extract all product names and prices', }); ``` **Available actions** | Action | Required fields | Description | | --------- | --------------------- | ---------------------------------------------------- | | `click` | `selector` or `value` | Click a button, link, or any element | | `type` | `selector`, `value` | Type text into an input or textarea | | `check` | `selector` or `value` | Check a checkbox | | `uncheck` | `selector` or `value` | Uncheck a checkbox | | `wait` | `duration` (ms) | Pause for a set number of milliseconds | | `scroll` | `to` (`0–100%`) | Scroll the page to a percentage of its height | | `forEach` | `observe` | Loop over every matched element and process each one | Use `selector` for a CSS selector or XPath. Use `value` for plain English — Spidra locates the element using AI. ```typescript theme={null} { type: 'click', selector: "button[data-testid='submit']" } // CSS selector { type: 'click', value: 'Accept all cookies button' } // plain English { type: 'type', selector: "input[name='q']", value: 'wireless headphones' } { type: 'wait', duration: 2000 } { type: 'scroll', to: '100%' } ``` ### forEach — loop over every element `forEach` finds a set of matching elements on the page and processes each one individually. Use it when you need to collect data from a list of items, paginate across pages, or click into each item's detail page. You don't need `forEach` if all the data fits on a single page — a plain `prompt` is simpler and works just as well. **Use `forEach` when:** * The list spans multiple pages and you need `pagination` * You need to click into each item's detail page (`navigate` mode) * You have 20+ items and want consistent per-item AI extraction (`itemPrompt`) #### inline mode Read each element's content directly without navigating away. Best for product cards, search results, and table rows. ```typescript theme={null} const job = await spidra.scrape.run({ urls: [ { url: 'https://books.toscrape.com', actions: [ { type: 'forEach', observe: 'Find all book cards in the product grid', mode: 'inline', captureSelector: 'article.product_pod', maxItems: 20, itemPrompt: 'Extract title, price, and star rating as JSON: {title, price, star_rating}', }, ], }, ], prompt: 'Return a clean JSON array of all books', output: 'json', }); ``` #### navigate mode Follow each element's link to its destination page and capture content there. Best for product listings where full details are only on individual pages. ```typescript theme={null} { type: 'forEach', observe: 'Find all book title links in the product grid', mode: 'navigate', captureSelector: 'article.product_page', maxItems: 10, waitAfterClick: 800, itemPrompt: 'Extract title, price, star rating, and availability as JSON', } ``` #### click mode Click each element, capture the content that appears (modal, drawer, or expanded section), then move on. Best for hotel room cards, FAQ accordions, or any UI where clicking reveals hidden content. ```typescript theme={null} { type: 'forEach', observe: 'Find all room type cards', mode: 'click', captureSelector: "[role='dialog']", maxItems: 8, waitAfterClick: 1200, itemPrompt: 'Extract room name, bed type, price per night, and amenities as JSON', } ``` #### Pagination After processing all elements on the current page, follow the next-page link and continue. ```typescript theme={null} { type: 'forEach', observe: 'Find all book title links', mode: 'navigate', maxItems: 40, pagination: { nextSelector: 'li.next > a', maxPages: 3, // 3 additional pages beyond the first }, } ``` `maxItems` applies across all pages combined. The loop stops when you hit `maxItems`, run out of elements, or reach `maxPages`. #### Per-element actions Run extra browser actions on each item after navigating or clicking into it, before content is captured. Useful for scrolling below the fold or expanding collapsed sections. ```typescript theme={null} { type: 'forEach', observe: 'Find all book title links', mode: 'navigate', captureSelector: 'article.product_page', maxItems: 5, waitAfterClick: 1000, actions: [ { type: 'scroll', to: '50%' }, ], itemPrompt: 'Extract title, price, and full description as JSON', } ``` #### itemPrompt vs top-level prompt Both are optional and serve different purposes: | | `itemPrompt` | `prompt` | | ------ | --------------------------------- | ----------------------------- | | Runs | During scraping, once per item | After all items are collected | | Sees | One item's content | All items combined | | Output | Feeds into the top-level `prompt` | `result.content` | Use `itemPrompt` to extract fields from each item individually. Use the top-level `prompt` to filter, sort, or reshape the combined output. They can be used together. ### Controlling how long run() waits By default `run()` waits until the job finishes, however long that takes — so a big crawl just works. If you'd rather cap the wait, pass a `timeout`; when it fires, a `SpidraTimeoutError` is thrown and the job keeps running server-side, so you can check it later with `get()` or cancel it. ```typescript theme={null} const controller = new AbortController(); const job = await spidra.scrape.run(params, { pollInterval: 3000, // ms between status checks (default: 3000) timeout: 600_000, // max wait in ms (default: none — wait until done) signal: controller.signal, // stop waiting without cancelling the job }); ``` Transient hiccups mid-wait — a 502 blip, a dropped connection, a rate limit — don't kill the wait; the SDK keeps polling unless several happen in a row. The same options work on `batch.run()` and `crawl.run()`. ## Batch scraping When you have a list of URLs to process — a product catalog, a set of listings, a pile of article links — batch is the right tool. Submit up to 50 URLs in one request and they all run in parallel. Each URL is a plain string (not an object). ### Scrape a list of URLs ```typescript theme={null} const batch = await spidra.batch.run({ urls: [ 'https://shop.example.com/product/1', 'https://shop.example.com/product/2', 'https://shop.example.com/product/3', ], prompt: 'Extract product name, price, and availability', output: 'json', useProxy: true, }); console.log(`${batch.completedCount}/${batch.totalUrls} succeeded`); for (const item of batch.items) { if (item.status === 'completed') console.log(item.url, item.result); if (item.status === 'failed') console.error(item.url, item.error); } ``` **Item statuses:** `pending` · `running` · `completed` · `failed` **Batch statuses:** `pending` · `running` · `completed` · `failed` · `cancelled` ### Submit now, check later If you don't want to hold a connection open while 50 pages scrape, submit the batch and come back whenever: ```typescript theme={null} const { batchId } = await spidra.batch.submit({ urls: ['https://example.com/1', 'https://example.com/2'], prompt: 'Extract the page title', }); // Later... const result = await spidra.batch.get(batchId); console.log(result.status, result.completedCount, '/', result.totalUrls); ``` ### Stream results as they finish Rather than waiting for the whole batch, `watch()` hands you each item the moment it completes — useful for writing results to a database as they arrive or updating a progress bar: ```typescript theme={null} const { batchId } = await spidra.batch.submit({ urls, prompt: 'Extract product data' }); const watcher = spidra.batch.watch(batchId); watcher.on('item', (item) => { console.log(item.url, item.status, item.result); // fires once per finished URL }); const final = await watcher.wait(); // resolves with the terminal batch state ``` Every item is delivered exactly once — including ones that already finished before you started watching. `watcher.stop()` stops listening without cancelling the batch. ### Retry failed items Re-queue only the items that failed — successful items are not re-run. ```typescript theme={null} const result = await spidra.batch.get(batchId); if (result.failedCount > 0) { const { retriedCount } = await spidra.batch.retry(batchId); console.log(`Retrying ${retriedCount} items`); } ``` ### Cancel a batch Stops all pending items and refunds credits for unprocessed work. ```typescript theme={null} const { cancelledItems, creditsRefunded } = await spidra.batch.cancel(batchId); console.log( `Cancelled ${cancelledItems} items, refunded ${creditsRefunded} credits`, ); ``` ### List past batches ```typescript theme={null} const { jobs, pagination } = await spidra.batch.list({ page: 1, limit: 20 }); for (const job of jobs) { console.log(job.uuid, job.status, `${job.completedCount}/${job.totalUrls}`); } ``` ## Crawling Crawling is different from scraping: you give it a starting URL and it discovers pages on its own, following links according to your instructions. Good for indexing a docs site, monitoring a competitor's blog, or building a structured dataset from an entire section of a site. ### Crawl a site ```typescript theme={null} const job = await spidra.crawl.run({ baseUrl: 'https://competitor.com/blog', crawlInstruction: 'Follow blog post links only, skip tag and category pages', transformInstruction: 'Extract the title, author, publish date, and a one-sentence summary', maxPages: 30, useProxy: true, }); for (const page of job.result) { console.log(page.url, page.data); } ``` **Parameters** | Parameter | Type | Default | Description | | ---------------------- | ---------- | ---------- | ------------------------------------------------------------------------------------ | | `baseUrl` | `string` | required | Starting URL for the crawl | | `crawlInstruction` | `string` | required | Which links to follow, in plain language | | `transformInstruction` | `string` | — | What to extract from each page. Omit for raw markdown mode (no AI, no token credits) | | `schema` | `object` | — | JSON Schema defining the exact output structure per page | | `maxPages` | `number` | `5` | Maximum pages to crawl (1–50) | | `maxDepth` | `number` | unlimited | Max link depth from the base URL. `0` = base URL only | | `includePaths` | `string[]` | — | URL path patterns to include, e.g. `["/blog/*"]` | | `excludePaths` | `string[]` | — | URL path patterns to skip, e.g. `["/tag/*"]` | | `allowSubdomains` | `boolean` | `false` | Follow links to subdomains of the base domain | | `crawlEntireDomain` | `boolean` | `false` | Follow any link on the same root domain | | `ignoreQueryParams` | `boolean` | `false` | Treat URLs differing only by query string as the same page | | `webhookUrl` | `string` | — | Receive a POST request for each processed page and on job completion | | `useProxy` | `boolean` | `false` | Route through a residential proxy | | `proxyCountry` | `string` | `"global"` | Two-letter country code, e.g. `"us"`. Requires `useProxy: true` | | `cookies` | `string` | — | Raw `Cookie` header string for authenticated sites | ### Raw content mode Omit both `transformInstruction` and `schema` to get the raw markdown of each page with no AI processing. No token credits are charged: ```typescript theme={null} const job = await spidra.crawl.run({ baseUrl: 'https://docs.example.com', crawlInstruction: 'Crawl all documentation pages', maxPages: 50, }); for (const page of job.result) { // page.data contains the raw markdown of each page console.log(page.url, page.data); } ``` ### Structured output with schema Use `schema` when you need every page to return the same fields in the same format: Use the [Spidra JSON Schema Generator](https://spidra.io/tools/json-schema-generator) to build and preview your schema visually before pasting it here. ```typescript theme={null} const job = await spidra.crawl.run({ baseUrl: 'https://example.com/jobs', crawlInstruction: 'Crawl all job listing pages', schema: { type: 'object', properties: { title: { type: 'string' }, location: { type: 'string' }, salary: { type: 'string' }, remote: { type: 'boolean' }, }, }, maxPages: 20, }); ``` ### Scoped crawling with path filters Use `includePaths` and `excludePaths` to keep crawls focused. Both accept glob-style patterns: ```typescript theme={null} const job = await spidra.crawl.run({ baseUrl: 'https://example.com', crawlInstruction: 'Crawl all documentation pages', transformInstruction: 'Extract the page title and main content', includePaths: ['/docs/*'], excludePaths: ['/docs/changelog/*', '/docs/legacy/*'], maxPages: 30, }); ``` ### Submit now, check later ```typescript theme={null} const { jobId } = await spidra.crawl.submit({ baseUrl: 'https://example.com/docs', crawlInstruction: 'Find all documentation pages', transformInstruction: 'Extract the page title and main content summary', maxPages: 50, }); // Poll manually const status = await spidra.crawl.get(jobId); // status.status: "waiting" | "active" | "running" | "completed" | "failed" | "cancelled" ``` ### Watch a crawl page-by-page A 50-page crawl can take a while. Instead of waiting for the whole thing, `watch()` streams each page to you the moment it's crawled: ```typescript theme={null} const { jobId } = await spidra.crawl.submit({ baseUrl: 'https://competitor.com/blog', crawlInstruction: 'Follow blog post links only', transformInstruction: 'Extract title, author, and publish date', maxPages: 50, }); const watcher = spidra.crawl.watch(jobId); watcher.on('page', (page) => { console.log(page.url, page.data); // fires once per crawled page }); watcher.on('error', (err) => console.error(err)); const final = await watcher.wait(); ``` Every page arrives exactly once — including pages crawled before you started watching — and the SDK only re-fetches page content when the crawl actually makes progress, so watching stays cheap. `watcher.stop()` stops listening without cancelling the crawl. ### Cancel a crawl Cancel a queued or running job at any time. Pages already processed are preserved: ```typescript theme={null} await spidra.crawl.cancel(jobId); // Retrieve whatever completed before cancellation const { pages } = await spidra.crawl.pages(jobId); for (const page of pages) { if (page.status === 'success') console.log(page.url, page.data); } ``` ### Download the raw HTML and Markdown `crawl.pages()` returns signed URLs for the raw HTML and Markdown of each crawled page. Links expire after **1 hour**. ```typescript theme={null} const { pages } = await spidra.crawl.pages(jobId); for (const page of pages) { console.log(page.url, page.status); // page.html — signed URL for the raw HTML snapshot // page.markdown — signed URL for the Markdown version } ``` ### Re-extract with a different prompt Crawled a site and want to pull out different information? You don't have to re-crawl. `crawl.extract()` runs a new AI pass over the already-crawled content and charges only transformation credits. ```typescript theme={null} const { jobId: newJobId } = await spidra.crawl.extract( sourceJobId, 'Extract only the product SKUs and prices as a CSV', ); const result = await spidra.crawl.get(newJobId); ``` ### History and stats ```typescript theme={null} // List past crawl jobs const { jobs, total, page, totalPages } = await spidra.crawl.history({ page: 1, limit: 10, }); // Total crawl job count for your account const { total: totalCrawls } = await spidra.crawl.stats(); ``` ## Logs Every API scrape job is logged automatically. Access your full history with optional filters. ### List and filter your logs ```typescript theme={null} const { logs, total } = await spidra.logs.list({ status: 'failed', // "success" | "failed" searchTerm: 'amazon.com', channel: 'api', // "api" | "playground" dateStart: '2024-01-01', dateEnd: '2024-12-31', page: 1, limit: 20, }); for (const log of logs) { console.log(log.urls[0]?.url, log.status, log.credits_used); } ``` **Filter parameters** | Parameter | Type | Description | | ------------ | ------------------------- | --------------------------------------------- | | `status` | `"success"` \| `"failed"` | Filter by outcome | | `searchTerm` | `string` | Search by URL or prompt | | `channel` | `string` | `"api"` or `"playground"` | | `dateStart` | `string` | ISO date — return logs on or after this date | | `dateEnd` | `string` | ISO date — return logs on or before this date | | `page` | `number` | Page number (default: 1) | | `limit` | `number` | Results per page (default: 20) | ### Get one log with its full output Fetch a single log entry, including the complete AI extraction result for that job. ```typescript theme={null} const log = await spidra.logs.get(logUuid); console.log(log.result_data); // full AI output for that job ``` ## Usage statistics Returns credit and request usage broken down by day or week. ```typescript theme={null} // Range options: "7d" | "30d" | "weekly" const rows = await spidra.usage.get('30d'); for (const row of rows) { console.log(row.date, row.requests, row.credits); } ``` | Range | Description | | ---------- | ------------------------------ | | `"7d"` | Last 7 days, one row per day | | `"30d"` | Last 30 days, one row per day | | `"weekly"` | Last 7 weeks, one row per week | ## Retries and reliability You don't have to write retry loops. Transient failures — network blips, 502/503/504 gateway errors — are retried automatically with exponential backoff, so a single hiccup never fails your call. Both knobs are configurable: ```typescript theme={null} const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY, maxRetries: 3, // retry attempts for transient failures (default: 3, 0 disables) backoffFactor: 500, // base ms — delay is backoffFactor * 2^attempt (default: 500) }); ``` The retry policy is designed so it can never double-charge you: 4xx client errors are never retried, and job submissions are only retried when the server explicitly rejected them — never on network errors or gateway timeouts, where the job may already have been queued. When the server sends a `Retry-After` hint, the SDK honors it instead of its own backoff. ## Error handling Every non-2xx response throws a typed error class. Catch the specific class you care about, or fall back to the base `SpidraError`. ```typescript theme={null} import { SpidraClient, SpidraError, SpidraAuthenticationError, SpidraInsufficientCreditsError, SpidraValidationError, SpidraRateLimitError, SpidraServerError, SpidraJobError, SpidraTimeoutError, } from 'spidra'; try { await spidra.scrape.run({ urls: [{ url: 'https://example.com' }], prompt: '...', }); } catch (err) { if (err instanceof SpidraAuthenticationError) { console.error('Invalid or missing API key'); // 401 } else if (err instanceof SpidraInsufficientCreditsError) { console.error('Out of credits — top up your account'); // 403 } else if (err instanceof SpidraValidationError) { console.error('Bad request:', err.errors); // 422 — one entry per problem } else if (err instanceof SpidraRateLimitError) { // 429 — the error tells you exactly how long to wait console.error(`Rate limited: ${err.remaining}/${err.limit} left, retry in ${err.retryAfterMs}ms`); } else if (err instanceof SpidraJobError) { // the job itself failed or was cancelled (not a transport error) console.error(`Job ${err.jobId} ${err.jobStatus}: ${err.message}`); } else if (err instanceof SpidraTimeoutError) { // your poll timeout elapsed — the job is still running server-side console.error(`Still running after ${err.timeoutMs}ms, check ${err.jobId} later`); } else if (err instanceof SpidraServerError) { console.error('Server error — already retried automatically'); // 5xx } else if (err instanceof SpidraError) { console.error(`API error ${err.status}: ${err.message}`); } } ``` **Error classes** | Class | Status | When | | -------------------------------- | ------ | ------------------------------------------------------------------------------------------- | | `SpidraAuthenticationError` | 401 | Missing or invalid API key | | `SpidraPaymentRequiredError` | 402 | Subscription payment overdue | | `SpidraInsufficientCreditsError` | 403 | Account has no remaining credits | | `SpidraNotFoundError` | 404 | Job, batch, or log does not exist | | `SpidraValidationError` | 422 | Request body failed validation — `err.errors` lists each problem | | `SpidraRateLimitError` | 429 | Too many requests — carries `err.limit`, `err.remaining`, `err.resetAt`, `err.retryAfterMs` | | `SpidraServerError` | 5xx | Unexpected error on Spidra's side | | `SpidraJobError` | — | The job itself failed or was cancelled — carries `err.jobId`, `err.jobStatus` | | `SpidraTimeoutError` | — | Your poll `timeout` elapsed; the job is still running — carries `err.jobId` | | `SpidraError` | other | Any other non-2xx response | All error classes expose `err.status` (the HTTP status code, or `0` for non-HTTP errors like job failures and timeouts) and `err.message`. API errors also carry `err.code` (a machine-readable identifier like `SERVICE_BUSY`) and `err.details` (the raw error body). ## Verifying webhooks Crawl jobs can push `crawl.page`, `crawl.completed`, and `crawl.failed` events to your `webhookUrl`. Spidra signs each delivery with HMAC-SHA256 in the `X-Spidra-Signature` header, and the SDK ships a helper so you never accept a forged event: ```typescript theme={null} import { verifySpidraWebhook } from 'spidra'; // Express example — verify against the RAW body, not the parsed JSON app.post('/webhooks/spidra', express.raw({ type: 'application/json' }), async (req, res) => { const valid = await verifySpidraWebhook( req.body, req.header('x-spidra-signature'), process.env.SPIDRA_WEBHOOK_SECRET!, ); if (!valid) return res.status(401).end(); const event = JSON.parse(req.body.toString()); if (event.event === 'crawl.page') { console.log('New page:', event.page.url); } res.status(200).end(); }); ``` The comparison is constant-time, and the helper works in Node, browsers, and edge runtimes. Always pass the raw request body, re-serializing parsed JSON produces different bytes and fails verification. ## AI agent integration Spidra works as a tool inside AI agent pipelines. Here is an example using the Vercel AI SDK with Claude: ```typescript theme={null} import { generateText, tool } from 'ai'; import { anthropic } from '@ai-sdk/anthropic'; import { SpidraClient } from 'spidra'; import { z } from 'zod'; const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY }); const result = await generateText({ model: anthropic('claude-opus-4-6'), maxSteps: 5, tools: { scrapeUrl: tool({ description: 'Fetch and extract structured data from a URL', parameters: z.object({ url: z.string().describe('The URL to scrape'), prompt: z.string().describe('What data to extract'), }), execute: async ({ url, prompt }) => { const job = await spidra.scrape.run({ urls: [{ url }], prompt }); return JSON.stringify(job.result.content); }, }), }, prompt: 'What are the top 3 trending repositories on GitHub today?', }); console.log(result.text); ``` Browse all official Spidra SDKs in one place. Official PHP SDK — idiomatic helpers, typed exceptions, and configurable polling. # SDKs Source: https://docs.spidra.io/sdks/overview Official Spidra SDKs for Node.js, Python, Go, PHP, Ruby, Java, .NET, Swift, Rust, Elixir, n8n, and MCP. All Spidra SDKs wrap the same REST API with idiomatic helpers for your language: authentication, async job polling, error handling, and typed responses. Pick your language and you are up and running in a few lines of code. All SDKs are open source at [github.com/spidra-io](https://github.com/spidra-io). ## Every SDK supports * Scrape, batch scrape, and crawl * Browser actions and forEach * Structured output with JSON Schema * Authenticated scraping with session cookies * Stealth mode and proxy geo-targeting * Async job submission and automatic polling ## Quick example Every SDK follows the same pattern. Here is a scrape across all languages: ```typescript Node.js theme={null} import { SpidraClient } from 'spidra' const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY }) const job = await spidra.scrape.run({ urls: [{ url: 'https://example.com/products' }], prompt: 'Extract all product names and prices', output: 'json', }) console.log(job.result.content) ``` ```python Python theme={null} import asyncio from spidra import AsyncSpidra, ScrapeParams, ScrapeUrl async def main(): spidra = AsyncSpidra(api_key="spd_YOUR_API_KEY") job = await spidra.scrape(ScrapeParams( urls=[ScrapeUrl(url="https://example.com/products")], prompt="Extract all product names and prices", output="json", )) print(job.content) asyncio.run(main()) ``` ```go Go theme={null} client := spidra.New(os.Getenv("SPIDRA_API_KEY")) job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{ URLs: []spidra.ScrapeURL{{URL: "https://example.com/products"}}, Prompt: "Extract all product names and prices", Output: "json", }) ``` ```php PHP theme={null} $spidra = new SpidraClient(getenv('SPIDRA_API_KEY')); $job = $spidra->scrape->run([ 'urls' => [['url' => 'https://example.com/products']], 'prompt' => 'Extract all product names and prices', 'output' => 'json', ]); ``` ```ruby Ruby theme={null} client = Spidra.new(ENV["SPIDRA_API_KEY"]) job = client.scrape.run( urls: [{ url: "https://example.com/products" }], prompt: "Extract all product names and prices", output: "json" ) puts job["result"]["content"] ``` `run()` submits the job and polls until it completes. Use `submit()` and `get()` separately if you need manual control over polling. ## Official SDKs Supports TypeScript natively. Works in Next.js, Express, Bun, and edge runtimes. LangChain and Vercel AI SDK tool support built in. Async-first with sync wrappers. Works in scripts, Django, Flask, FastAPI, and Jupyter notebooks. Typed structs, idiomatic error handling, zero external dependencies. Standard library only. Requires PHP 8.1+ and Guzzle 7. Typed exceptions and configurable polling timeouts. Pure stdlib, no external dependencies. Works in Rails, Sinatra, and plain scripts. Idiomatic pattern matching, OTP-ready. Works with Phoenix and plain Mix projects. Fully async with `Task`/`await`. Typed exceptions and JSON schema support. Requires .NET 8+. Native `async/await` concurrency. Works on iOS, macOS, tvOS, watchOS, and server-side Swift. Java 17+. `CompletableFuture`-based async, builder pattern, no extra HTTP dependencies. Built on `tokio` and `reqwest`. Native async/await, returns `Result` on every call. ## Workflow integrations Native node for n8n workflows. Trigger scrapes, batch jobs, and crawls as steps in any automation. Give any AI assistant live web access as native tools. Works with Claude Code, Claude Desktop, Cursor, Windsurf, and VS Code. # PHP Source: https://docs.spidra.io/sdks/php Official PHP SDK for the Spidra web scraping API. Handles job submission, status polling, and error mapping. Requires PHP 8.1+ with Guzzle. The PHP SDK wraps the Spidra API with idiomatic PHP helpers so you're not writing raw HTTP calls and hand-rolling polling loops. It handles job submission, status polling, error mapping to typed exceptions, and everything in between. ## Installation ```bash theme={null} composer require spidra/spidra-php ``` Requires PHP 8.1+ and Guzzle 7. Once Composer is done, you're good to go — no extra setup. Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Store it as an environment variable. Never hardcode it. ## Getting started ```php theme={null} use Spidra\SpidraClient; $spidra = new SpidraClient(getenv('SPIDRA_API_KEY')); ``` From here you access everything through `$spidra->scrape`, `$spidra->batch`, `$spidra->crawl`, `$spidra->logs`, and `$spidra->usage`. ## Scraping The scraper accepts up to three URLs per request and processes them in parallel. You can pass a plain extraction prompt, a full JSON schema, per-URL browser actions, or any mix of those. The simplest path is `run()` — it submits the job and blocks until it finishes, then returns the result: ```php theme={null} $job = $spidra->scrape->run([ 'urls' => [['url' => 'https://example.com/pricing']], 'prompt' => 'Extract all pricing plans with name, price, and included features', 'output' => 'json', ]); print_r($job['content']); // ['plans' => [['name' => 'Starter', 'price' => '$9/mo', ...], ...]] ``` If you'd rather fire and move on, `submit()` returns a `jobId` immediately. You can then call `get()` whenever you're ready to check: ```php theme={null} ['jobId' => $jobId] = $spidra->scrape->submit([ 'urls' => [['url' => 'https://example.com']], 'prompt' => 'Extract the main headline', ]); // Later... $status = $spidra->scrape->get($jobId); if ($status['status'] === 'completed') { echo $status['content']; } ``` Job statuses move through: `waiting` → `active` → `completed` (or `failed`). ### Scrape parameters | Parameter | Type | Description | | -------------------- | -------- | ------------------------------------------------------------------ | | `urls` | `array` | Up to 3 URLs. Each entry is `['url' => '...', 'actions' => [...]]` | | `prompt` | `string` | What to extract. Written in plain English | | `output` | `string` | `"markdown"` (default) or `"json"` | | `schema` | `array` | JSON Schema — forces a specific shape when using `output: "json"` | | `useProxy` | `bool` | Route through a residential proxy | | `proxyCountry` | `string` | Two-letter country code: `"us"`, `"de"`, `"jp"`, etc. | | `extractContentOnly` | `bool` | Strip nav, ads, and boilerplate before the AI sees the page | | `screenshot` | `bool` | Capture a viewport screenshot | | `fullPageScreenshot` | `bool` | Capture a full-page (scrolled) screenshot | | `cookies` | `string` | Raw `Cookie` header string for pages behind a login | ### Enforcing an exact output shape Without a schema, the AI extracts what it finds. With a schema, missing fields come back as `null` rather than guessed values — useful when the output feeds a database or a typed pipeline downstream. ```php theme={null} $job = $spidra->scrape->run([ 'urls' => [['url' => 'https://jobs.example.com/senior-engineer']], 'prompt' => 'Extract the job listing details', 'output' => 'json', 'schema' => [ 'type' => 'object', 'required' => ['title', 'company', 'remote'], 'properties' => [ 'title' => ['type' => 'string'], 'company' => ['type' => 'string'], 'remote' => ['type' => ['boolean', 'null']], 'salary_min' => ['type' => ['number', 'null']], 'skills' => ['type' => 'array', 'items' => ['type' => 'string']], ], ], ]); ``` ### Scraping geo-restricted content Some sites serve different prices or content depending on where you're browsing from. Set `useProxy` and `proxyCountry` to route through a residential IP in that country: ```php theme={null} $job = $spidra->scrape->run([ 'urls' => [['url' => 'https://www.amazon.de/gp/bestsellers']], 'prompt' => 'List the top 10 products with name and price', 'useProxy' => true, 'proxyCountry' => 'de', ]); ``` Supported country codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing without pinning to a specific country. ### Scraping pages behind a login If the page requires a session, pass your cookies as a raw header string. The easiest way to get this is to log in through your browser's devtools, then copy the `Cookie` header from any authenticated request. ```php theme={null} $job = $spidra->scrape->run([ 'urls' => [['url' => 'https://app.example.com/dashboard']], 'prompt' => 'Extract the monthly revenue and active user count', 'cookies' => 'session=abc123; auth_token=xyz789', ]); ``` ### Browser actions Sometimes you need to interact with the page before extraction — dismiss a cookie banner, type into a search box, scroll to load lazy content. Pass an `actions` array inside the URL entry and they'll run in order before the AI sees the page: ```php theme={null} $job = $spidra->scrape->run([ 'urls' => [ [ 'url' => 'https://example.com/products', 'actions' => [ ['type' => 'click', 'selector' => '#accept-cookies'], ['type' => 'wait', 'duration' => 1000], ['type' => 'scroll', 'to' => '80%'], ], ], ], 'prompt' => 'Extract all product names and prices visible on the page', ]); ``` For `selector` you can pass a CSS selector or XPath. If you'd rather describe the element in plain English, use `value` — Spidra will locate it with AI. | Action | What it does | | --------- | ------------------------------------------------------------------ | | `click` | Click any element — use `selector` for CSS, `value` for plain text | | `type` | Type into an input or textarea | | `check` | Check a checkbox | | `uncheck` | Uncheck a checkbox | | `wait` | Pause for `duration` milliseconds | | `scroll` | Scroll to a percentage of the page height (e.g. `"80%"`) | | `forEach` | Loop over every matched element and extract from each one | ### Controlling how long run() waits By default `run()` polls every 3 seconds and gives up after 120 seconds. You can override both: ```php theme={null} $job = $spidra->scrape->run($params, [ 'pollInterval' => 5, // seconds between checks 'timeout' => 60, // throw after this many seconds if still running ]); ``` The same options work on `batch->run()` and `crawl->run()`. ## Batch scraping When you have a list of URLs to process, batch is the right tool. You can submit up to 50 URLs in a single request and they all run in parallel. Unlike the scraper, each URL here is a plain string — there's no per-URL actions support. ```php theme={null} $batch = $spidra->batch->run([ 'urls' => [ 'https://shop.example.com/product/1', 'https://shop.example.com/product/2', 'https://shop.example.com/product/3', ], 'prompt' => 'Extract product name, price, and whether it is in stock', 'output' => 'json', ]); echo $batch['completedCount'] . '/' . $batch['totalUrls'] . " completed\n"; foreach ($batch['items'] as $item) { if ($item['status'] === 'completed') { print_r($item['result']); } else { echo "Failed: {$item['url']} — {$item['error']}\n"; } } ``` Each item in `items` moves through `pending` → `running` → `completed` (or `failed`). The batch itself follows the same lifecycle, plus a `cancelled` state if you stop it early. If you don't want to wait for the whole batch to finish, use `submit()` and `get()` separately: ```php theme={null} ['batchId' => $batchId] = $spidra->batch->submit([ 'urls' => ['https://example.com/1', 'https://example.com/2'], 'prompt' => 'Extract the page title and meta description', ]); // Come back later $result = $spidra->batch->get($batchId); echo "{$result['completedCount']} of {$result['totalUrls']} done\n"; ``` ### Retrying failures and cancelling If some items fail (transient network errors, timeouts), you can retry just those without re-running the ones that already succeeded: ```php theme={null} if ($batch['failedCount'] > 0) { $retry = $spidra->batch->retry($batchId); echo "Retrying {$retry['retriedCount']} failed items\n"; } ``` To stop a running batch and get credits back for anything that hasn't started yet: ```php theme={null} $result = $spidra->batch->cancel($batchId); echo "Cancelled {$result['cancelledItems']} items — {$result['creditsRefunded']} credits refunded\n"; ``` To look through past batches: ```php theme={null} $page = $spidra->batch->list(1, 20); // page, limit foreach ($page['jobs'] as $job) { echo "{$job['uuid']} {$job['status']} — {$job['completedCount']}/{$job['totalUrls']}\n"; } ``` ## Crawling Crawling is different from scraping — you give it a starting URL and it discovers and processes pages on its own, following links according to your instructions. Good for indexing a docs site, monitoring a competitor's blog, or building a structured dataset from an entire section of a site. ```php theme={null} $job = $spidra->crawl->run([ 'baseUrl' => 'https://competitor.com/blog', 'crawlInstruction' => 'Follow links to blog posts only — skip tag pages, category pages, and the homepage', 'transformInstruction' => 'Extract the post title, author name, publish date, and a one-sentence summary', 'maxPages' => 50, 'useProxy' => true, ]); foreach ($job['result'] as $page) { echo $page['url'] . "\n"; print_r($page['data']); } ``` `crawlInstruction` tells the crawler which links to follow. `transformInstruction` tells the AI what to extract from each page it visits. `maxPages` is a safety cap — the crawl stops once it hits that number. The same `useProxy`, `proxyCountry`, and `cookies` options from the scraper work here too. Just like scraping, you can fire-and-forget with `submit()` and poll with `get()`: ```php theme={null} ['jobId' => $jobId] = $spidra->crawl->submit([ 'baseUrl' => 'https://example.com/docs', 'crawlInstruction' => 'Follow all documentation pages', 'transformInstruction' => 'Extract the page title and a short summary of the content', 'maxPages' => 50, ]); $status = $spidra->crawl->get($jobId); // status moves through: waiting → active → running → completed (or failed) ``` ### Downloading the raw content Once a crawl completes, you can fetch signed URLs to download the raw HTML and Markdown for every page that was crawled. These links expire after an hour: ```php theme={null} $result = $spidra->crawl->pages($jobId); foreach ($result['pages'] as $page) { // $page['html_url'] — download the raw HTML // $page['markdown_url'] — download the cleaned Markdown echo $page['url'] . ' — ' . $page['status'] . "\n"; } ``` ### Re-extracting with a different prompt If you crawled a site and want to pull out different information — say you originally extracted titles and summaries, but now you need prices — you don't have to re-crawl. `extract()` runs a new AI pass over the already-crawled content and charges only transformation credits: ```php theme={null} $result = $spidra->crawl->extract( $completedJobId, 'Extract only product SKUs and prices as structured JSON' ); // This creates a new job — poll it like any other $extracted = $spidra->crawl->get($result['jobId']); ``` ### Browsing your crawl history ```php theme={null} $history = $spidra->crawl->history(1, 10); echo "Total crawl jobs: {$history['total']}\n"; $stats = $spidra->crawl->stats(); echo "All-time: {$stats['total']}\n"; ``` ## Logs Every scrape request your API key makes gets logged automatically. You can filter by status, URL, date range, or where it came from (API vs playground): ```php theme={null} $result = $spidra->logs->list([ 'status' => 'failed', 'searchTerm' => 'amazon.com', 'dateStart' => '2024-01-01', 'dateEnd' => '2024-12-31', 'page' => 1, 'limit' => 20, ]); foreach ($result['logs'] as $log) { echo $log['urls'][0]['url'] . ' — ' . $log['status'] . ' (' . $log['credits_used'] . ' credits)' . "\n"; } ``` To fetch the full details of a single log entry, including the AI extraction output: ```php theme={null} $log = $spidra->logs->get($logUuid); print_r($log['result_data']); ``` ## Usage statistics Check how many requests and credits your account has used over a given period: ```php theme={null} $rows = $spidra->usage->get('30d'); // "7d" | "30d" | "weekly" foreach ($rows as $row) { echo "{$row['date']}: {$row['requests']} requests, {$row['credits']} credits\n"; } ``` `"7d"` gives one row per day for the last week. `"30d"` gives the last month. `"weekly"` gives one row per week for the last seven weeks. ## Error handling Every API error is mapped to a typed exception, so you can catch exactly what you care about and ignore the rest: ```php theme={null} use Spidra\Exceptions\SpidraException; use Spidra\Exceptions\AuthenticationException; use Spidra\Exceptions\InsufficientCreditsException; use Spidra\Exceptions\RateLimitException; use Spidra\Exceptions\ServerException; try { $job = $spidra->scrape->run([ 'urls' => [['url' => 'https://example.com']], 'prompt' => 'Extract the main headline', ]); } catch (AuthenticationException $e) { // Bad or missing API key } catch (InsufficientCreditsException $e) { // Account is out of credits — time to top up } catch (RateLimitException $e) { // Slow down — you're hitting limits } catch (ServerException $e) { // Something went wrong on Spidra's side — retry is usually safe } catch (SpidraException $e) { // Catch-all for anything else echo "Error {$e->getCode()}: {$e->getMessage()}\n"; } ``` | Exception | HTTP | Meaning | | ------------------------------ | ---- | ------------------------------------ | | `AuthenticationException` | 401 | The API key is missing or invalid | | `InsufficientCreditsException` | 403 | No credits remaining on the account | | `RateLimitException` | 429 | Too many requests — back off | | `ServerException` | 500 | Unexpected server-side error | | `SpidraException` | any | Base class for all Spidra exceptions | All exceptions expose `getCode()` for the HTTP status and `getMessage()` for a human-readable explanation. Official Node.js / TypeScript SDK — works in Next.js, Express, Bun, and edge runtimes. Official Go SDK — typed structs, idiomatic errors, zero external dependencies. # Python Source: https://docs.spidra.io/sdks/python Official Python SDK for the Spidra web scraping API. The Python SDK wraps the Spidra API so you're not writing raw HTTP calls and polling loops yourself. It handles job submission, status polling, retry logic, and error mapping to typed exceptions. ## Installation ```bash theme={null} pip install spidra ``` Requires Python 3.9 or higher. Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**. Store it as an environment variable. Never hardcode it. ## Getting started ```python theme={null} from spidra import Spidra spidra = Spidra(api_key="spd_YOUR_API_KEY") ``` Name the instance `spidra`, `client`, or whatever fits your codebase — the method names stay the same. If you're inside an existing async context (FastAPI, asyncio, Jupyter notebook), use `AsyncSpidra` instead and `await` the calls. The method signatures are identical. ## Scraping The scraper accepts up to three URLs per request and processes them in parallel. You can pass a URL string directly, or a `ScrapeParams` object for full control. The simplest call: ```python theme={null} from spidra import Spidra spidra = Spidra(api_key="spd_YOUR_API_KEY") result = spidra.scrape( "https://example.com/pricing", prompt="Extract all pricing plans with name, price, and included features", output="json", ) print(result.content) # {"plans": [{"name": "Starter", "price": "$9/mo", ...}]} ``` If you'd rather fire and move on, `start_scrape()` returns a job ID immediately. You can then call `get_scrape()` whenever you're ready to check: ```python theme={null} queued = spidra.start_scrape( "https://example.com", prompt="Extract the main headline", ) # Later... status = spidra.get_scrape(queued.job_id) if status.status == "completed": print(status.result.content) ``` Job statuses move through: `queued` → `waiting` → `active` → `completed` (or `failed`). ### Scrape parameters | Parameter | Type | Description | | ---------------------- | ---- | ---------------------------------------------------------------------- | | `urls` | list | Up to 3 `ScrapeUrl` objects. Each takes a `url` and optional `actions` | | `prompt` | str | What to extract, written in plain English | | `output` | str | `"markdown"` (default) or `"json"` | | `schema` | dict | JSON Schema that forces a specific output shape | | `use_proxy` | bool | Route through a residential proxy | | `proxy_country` | str | Two-letter country code: `"us"`, `"de"`, `"jp"`, etc. | | `extract_content_only` | bool | Strip nav, ads, and boilerplate before the AI sees the page | | `screenshot` | bool | Capture a viewport screenshot | | `full_page_screenshot` | bool | Capture a full-page scrolled screenshot | | `cookies` | str | Raw `Cookie` header string for pages behind a login | ### Enforcing an exact output shape Without a schema the AI extracts what it finds. With a schema, missing fields come back as `None` rather than guessed values, which matters when the output feeds a database or a typed pipeline downstream: Use the [Spidra JSON Schema Generator](https://spidra.io/tools/json-schema-generator) to build and preview your schema visually before pasting it here. ```python theme={null} from spidra import ScrapeParams, ScrapeUrl result = spidra.scrape(ScrapeParams( urls=[ScrapeUrl(url="https://jobs.example.com/senior-engineer")], prompt="Extract the job listing details", output="json", schema={ "type": "object", "required": ["title", "company", "remote"], "properties": { "title": {"type": "string"}, "company": {"type": "string"}, "remote": {"type": ["boolean", "null"]}, "salary_min": {"type": ["number", "null"]}, "skills": {"type": "array", "items": {"type": "string"}}, }, }, )) ``` Define every field you want extracted. An untyped `object` with no `properties` (or an array of them) gives the AI nothing to fill in, so those members come back empty. ### Enforcing shape with Pydantic If you already model your data with [Pydantic](https://docs.pydantic.dev), skip the JSON Schema entirely — pass the model itself (class or instance) and the SDK converts it for you: ```python theme={null} from pydantic import BaseModel class JobListing(BaseModel): title: str company: str remote: bool | None = None skills: list[str] = [] result = spidra.scrape( "https://jobs.example.com/senior-engineer", prompt="Extract the job listing details", output="json", schema=JobListing, ) listing = JobListing.model_validate(result.content) # validated, typed access ``` The same works on `batch_scrape()` and `crawl()` (applied per page). Pydantic stays optional — install it with `pip install spidra[pydantic]` only if you use this. Both v2 and v1 models are supported. ### Scraping geo-restricted content Some sites serve different prices or content depending on where you're browsing from. Set `use_proxy=True` and a `proxy_country` code to route through a residential IP in that country: ```python theme={null} result = spidra.scrape( "https://www.amazon.de/gp/bestsellers", prompt="List the top 10 products with name and price", use_proxy=True, proxy_country="de", ) ``` Supported country codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing without pinning to a specific country. ### Scraping pages behind a login If the page requires a session, pass your cookies as a raw header string. The easiest way to get this is to log in through your browser, open devtools, and copy the `Cookie` header from any authenticated request: ```python theme={null} result = spidra.scrape( "https://app.example.com/dashboard", prompt="Extract the monthly revenue and active user count", cookies="session=abc123; auth_token=xyz789", ) ``` ### Browser actions Sometimes you need to interact with the page before extraction — dismiss a cookie banner, type into a search box, scroll to load lazy content. Pass an `actions` list inside the `ScrapeUrl` and they run in order before the AI sees the page: ```python theme={null} from spidra import ScrapeUrl, BrowserAction result = spidra.scrape( ScrapeUrl( url="https://example.com/products", actions=[ BrowserAction(type="click", selector="#accept-cookies"), BrowserAction(type="wait", duration=1000), BrowserAction(type="scroll", to="80%"), ], ), prompt="Extract all product names and prices visible on the page", ) ``` For `selector` you can pass a CSS selector or XPath. If you'd rather describe the element in plain English, use `value` and Spidra will locate it with AI. | Action | What it does | | --------- | ------------------------------------------------------------------ | | `click` | Click any element — use `selector` for CSS, `value` for plain text | | `type` | Type into an input or textarea | | `check` | Check a checkbox | | `uncheck` | Uncheck a checkbox | | `wait` | Pause for `duration` milliseconds | | `scroll` | Scroll to a percentage of the page height, e.g. `"80%"` | | `forEach` | Loop over every matched element and extract from each one | ### Controlling how long scrape() waits By default `scrape()` polls every 3 seconds and waits until the job finishes, however long that takes. If you'd rather cap the wait, pass a `timeout` in seconds — when it fires, `SpidraTimeoutError` is raised and the job keeps running server-side, so you can check it later with `get_scrape()` or cancel it: ```python theme={null} result = spidra.scrape( "https://example.com", prompt="...", poll_interval=5, timeout=300, ) ``` Transient hiccups mid-wait — a 502 blip, a dropped connection, a rate limit — don't kill the wait; the SDK keeps polling unless several happen in a row. The same options work on `batch_scrape()` and `crawl()`. ## Batch scraping When you have a list of URLs to process, batch is the right tool. You can submit up to 50 URLs in a single request and they all run in parallel. ```python theme={null} batch = spidra.batch_scrape( ["https://shop.example.com/product/1", "https://shop.example.com/product/2"], prompt="Extract product name, price, and whether it is in stock", output="json", ) print(f"{batch.completed_count}/{batch.total_urls} completed") for item in batch.items: if item.status == "completed": print(item.url, item.result) else: print(f"Failed: {item.url} — {item.error}") ``` Each item moves through `pending` → `running` → `completed` (or `failed`). If you don't want to wait for the whole thing to finish, use `start_batch_scrape()` and `get_batch_scrape()` separately: ```python theme={null} queued = spidra.start_batch_scrape( ["https://example.com/1", "https://example.com/2"], prompt="Extract the page title and meta description", ) # Come back later result = spidra.get_batch_scrape(queued.batch_id) print(f"{result.completed_count} of {result.total_urls} done") ``` ### Retrying failures and cancelling ```python theme={null} if batch.failed_count > 0: spidra.retry_batch_scrape(queued.batch_id) # Cancel a running batch and refund unprocessed credits response = spidra.cancel_batch_scrape(batch_id) print(f"Cancelled {response.cancelled_items} items, refunded {response.credits_refunded} credits") ``` To look through past batches: ```python theme={null} page = spidra.list_batch_scrapes(page=1, limit=20) for job in page.jobs: print(job.uuid, job.status, f"{job.completed_count}/{job.total_urls}") ``` ## Crawling Crawling is different from scraping. You give it a starting URL and it discovers and processes pages on its own, following links according to your instructions. Good for indexing a docs site, monitoring a competitor's blog, or building a structured dataset from an entire section of a site. ```python theme={null} job = spidra.crawl( "https://competitor.com/blog", crawl_instruction="Follow links to blog posts only, skip tag pages and the homepage", transform_instruction="Extract the post title, author name, publish date, and a one-sentence summary", max_pages=30, ) for page in job.result: print(page.url, page.data) ``` `crawl_instruction` tells the crawler which links to follow. `transform_instruction` tells the AI what to extract from each page. By default the call waits until the crawl finishes — pass `timeout=` to bound the wait (the job keeps running server-side if it fires). ### Raw content mode Omit both `transform_instruction` and `schema` to get the raw page content without any AI processing. Each page's `data` field contains the plain markdown of that page — no token credits are charged: ```python theme={null} job = spidra.crawl( "https://docs.example.com", crawl_instruction="Crawl all documentation pages", max_pages=50, ) for page in job.result: # page.data contains the raw markdown of each page print(page.url, page.data[:200]) ``` This is useful when you want to feed the content into your own AI pipeline. ### Structured output with schema When you need every page to return the same fields in the same format, use `schema`. The AI returns JSON matching it exactly for every page: Use the [Spidra JSON Schema Generator](https://spidra.io/tools/json-schema-generator) to build and preview your schema visually before pasting it here. ```python theme={null} from spidra import CrawlParams job = spidra.crawl(CrawlParams( base_url="https://example.com/jobs", crawl_instruction="Crawl all job listing pages", schema={ "type": "object", "properties": { "title": {"type": "string"}, "location": {"type": "string"}, "salary": {"type": "string"}, "remote": {"type": "boolean"}, }, }, max_pages=20, )) for page in job.result: print(page.data) # {"title": "...", "location": "...", ...} ``` ### Scoped crawling with path filters Use `include_paths` and `exclude_paths` to keep crawls focused on the content you actually need. Both accept glob-style patterns: ```python theme={null} job = spidra.crawl(CrawlParams( base_url="https://example.com", crawl_instruction="Crawl all documentation pages", transform_instruction="Extract the page title and main content", include_paths=["/docs/*"], exclude_paths=["/docs/changelog/*", "/docs/legacy/*"], max_pages=30, )) ``` ### Crawl parameters | Parameter | Type | Default | Description | | ----------------------- | ---------- | --------------------------------- | -------------------------------------------------------------------- | | `base_url` | str | required | Starting URL for the crawl | | `crawl_instruction` | str | `"Find all pages on the website"` | Which links to follow, in plain language | | `transform_instruction` | str | — | What to extract from each page. Omit for raw markdown mode | | `schema` | dict | — | JSON Schema defining the exact output structure per page | | `max_pages` | int | `5` | Maximum pages to crawl (1–50) | | `max_depth` | int | unlimited | Max link depth from the base URL. `0` = base URL only | | `include_paths` | list\[str] | — | URL path patterns to include, e.g. `["/blog/*"]` | | `exclude_paths` | list\[str] | — | URL path patterns to skip, e.g. `["/tag/*"]` | | `allow_subdomains` | bool | `False` | Follow links to subdomains of the base domain | | `crawl_entire_domain` | bool | `False` | Follow any link on the same root domain | | `ignore_query_params` | bool | `False` | Treat URLs differing only by query string as the same page | | `webhook_url` | str | — | Receive a POST request for each processed page and on job completion | | `use_proxy` | bool | `False` | Route through a residential proxy | | `proxy_country` | str | `"global"` | Two-letter country code. Requires `use_proxy=True` | | `cookies` | str | — | Cookie header string for authenticated crawls | ### Submitting without waiting Just like scraping, you can fire-and-forget with `start_crawl()` and poll with `get_crawl()`: ```python theme={null} queued = spidra.start_crawl( "https://example.com/docs", crawl_instruction="Follow all documentation pages", transform_instruction="Extract the page title and a short summary", max_pages=50, ) # Poll manually status = spidra.get_crawl(queued.job_id) if status.status == "completed": for page in status.result: print(page.url, page.data) ``` ### Cancelling a crawl Cancel a queued or running job at any time. Pages already processed are kept: ```python theme={null} response = spidra.cancel_crawl(job_id) print(response.status) # "cancelled" # Retrieve whatever was processed before cancellation pages = spidra.crawl_pages(job_id) for page in pages.pages: if page.status == "success": print(page.url, page.data) ``` ### Downloading the raw HTML and Markdown Once a crawl completes, `crawl_pages()` returns signed download URLs for the raw HTML and Markdown of every page. These links expire after one hour: ```python theme={null} response = spidra.crawl_pages(job_id) for page in response.pages: print(page.url, page.status) # page.html — signed URL for the raw HTML snapshot # page.markdown — signed URL for the Markdown version ``` ### Re-extracting with a different prompt If you crawled a site and want to pull out different information, you don't have to re-crawl. `crawl_extract()` runs a new AI pass over the already-crawled content and charges only transformation credits: ```python theme={null} queued = spidra.crawl_extract( completed_job_id, "Extract only product SKUs and prices as structured JSON", ) result = spidra.get_crawl(queued.job_id) ``` ### Browsing your crawl history ```python theme={null} response = spidra.crawl_history(page=1, limit=10) print(f"Total crawl jobs: {response.total}") stats = spidra.crawl_stats() print(f"All-time: {stats.total}") ``` ## Watching jobs (streaming results) A 50-page crawl can take a while. Instead of waiting for the whole thing, `watch_crawl()` yields each page the moment it's crawled — perfect for writing results to a database as they arrive or updating a progress bar: ```python theme={null} queued = spidra.start_crawl( "https://competitor.com/blog", crawl_instruction="Follow blog post links only", transform_instruction="Extract title, author, and publish date", max_pages=50, ) for page in spidra.watch_crawl(queued.job_id): print(page.url, page.data) # fires once per crawled page ``` Batches work the same way — `watch_batch()` yields each item as it finishes (completed or failed): ```python theme={null} queued = spidra.start_batch_scrape(urls, prompt="Extract product data") for item in spidra.watch_batch(queued.batch_id): print(item.url, item.status, item.result) ``` On `AsyncSpidra` these are async generators — same names, just `async for`. Every page/item is yielded exactly once, including ones that finished before you started watching, and page content is only re-fetched when the crawl actually makes progress, so watching stays cheap. The loop ends when the job completes or is cancelled, raises `SpidraJobFailedError` if it fails, and breaking out early never cancels the job — use `cancel_crawl()` for that. ## Logs Every scrape request your API key makes gets logged automatically. You can filter by status, URL, date range, or where it came from: ```python theme={null} response = spidra.scrape_logs( status="failed", search_term="amazon.com", start_date="2024-01-01", end_date="2024-12-31", page=1, limit=20, ) for log in response.logs: print(log.urls[0].get("url"), log.status, log.credits_used) ``` To fetch the full details of a single log entry including the AI extraction output: ```python theme={null} log = spidra.get_scrape_log(log_uuid) print(log.result_data) ``` ## Usage statistics Check how many requests and credits your account has used over a given period: ```python theme={null} rows = spidra.usage("30d") # "7d" | "30d" | "weekly" for row in rows: print(row.date, row.requests, row.credits) ``` ## Retries and reliability You don't have to write retry loops. Transient failures — network blips, 502/503/504 gateway errors — are retried automatically with exponential backoff, so a single hiccup never fails your call. Both knobs are configurable: ```python theme={null} spidra = Spidra( api_key="spd_YOUR_API_KEY", max_retries=3, # retry attempts for transient failures (default: 3, 0 disables) backoff_factor=1.0, # base seconds — delay is backoff_factor * 2**(attempt-1) ) ``` The retry policy is designed so it can never double-charge you: 4xx client errors are never retried, and job submissions are only retried when the server explicitly rejected them — never on network errors or gateway timeouts, where the job may already have been queued. When the server sends a `Retry-After` hint, the SDK honors it instead of its own backoff. ## Error handling Every API error is mapped to a typed exception class, so you can catch exactly what you care about and let the rest bubble up: ```python theme={null} from spidra import ( SpidraError, SpidraAuthenticationError, SpidraForbiddenError, SpidraValidationError, SpidraRateLimitError, SpidraServerError, SpidraJobFailedError, SpidraTimeoutError, ) try: result = spidra.scrape("https://example.com", prompt="Extract the main headline") except SpidraAuthenticationError: # 401: Missing or invalid API key pass except SpidraForbiddenError: # 403: Monthly credit limit reached pass except SpidraValidationError as e: # 422: Bad request body — e.errors lists each problem print(e.errors) except SpidraRateLimitError as e: # 429: Too many requests — metadata tells you exactly how long to wait print(f"Rate limited. {e.remaining}/{e.limit} left, retry in {e.retry_after}s") except SpidraJobFailedError as e: # The job itself failed or was cancelled (not a transport error) print(f"Job {e.job_id} {e.job_status}: {e.message}") except SpidraTimeoutError as e: # Your poll timeout elapsed — the job is still running server-side print(f"Still running after {e.timeout_seconds}s, check {e.job_id} later") except SpidraServerError as e: # 5xx: Something went wrong on Spidra's side (already retried automatically) print(f"Server error ({e.status}): {e.message}") except SpidraError as e: # Catch-all for anything else print(f"API error {e.status}: {e.message}") ``` | Exception | Status | When | | ---------------------------- | ------ | ----------------------------------------------------------------------------------- | | `SpidraAuthenticationError` | 401 | Missing or invalid API key | | `SpidraPaymentRequiredError` | 402 | Subscription payment overdue | | `SpidraForbiddenError` | 403 | Monthly credit limit reached | | `SpidraNotFoundError` | 404 | Job, batch, or log does not exist | | `SpidraValidationError` | 422 | Request body failed validation — `e.errors` lists each problem | | `SpidraRateLimitError` | 429 | Too many requests — carries `e.limit`, `e.remaining`, `e.reset_at`, `e.retry_after` | | `SpidraServerError` | 5xx | Unexpected server-side error | | `SpidraJobFailedError` | — | The job itself failed or was cancelled — carries `e.job_id`, `e.job_status` | | `SpidraTimeoutError` | — | Your poll `timeout` elapsed; the job is still running — carries `e.job_id` | | `SpidraError` | any | Base class for all Spidra exceptions | All exceptions expose `.status` (the HTTP status code, or `0` for non-HTTP errors like job failures and timeouts) and `.message`. API errors also carry `.code` (a machine-readable identifier like `SERVICE_BUSY`) and `.details` (the raw error body). ## Verifying webhooks Crawl jobs can push `crawl.page`, `crawl.completed`, and `crawl.failed` events to your `webhook_url`. Spidra signs each delivery with HMAC-SHA256 in the `X-Spidra-Signature` header, and the SDK ships a helper so you never accept a forged event: ```python theme={null} import json from spidra import verify_webhook # FastAPI example — verify against the RAW body, not the parsed JSON @app.post("/webhooks/spidra") async def spidra_webhook(request: Request): raw = await request.body() if not verify_webhook(raw, request.headers.get("x-spidra-signature"), WEBHOOK_SECRET): raise HTTPException(status_code=401) event = json.loads(raw) if event["event"] == "crawl.page": print("New page:", event["page"]["url"]) return {"ok": True} ``` The helper uses only the standard library and compares in constant time. Always pass the raw request body — re-serialising parsed JSON produces different bytes and fails verification. Official Ruby SDK — pure stdlib, no external dependencies. Works in Rails, Sinatra, and scripts. Official Elixir SDK — idiomatic pattern matching, OTP-ready, works with Phoenix and plain Mix projects. # Ruby Source: https://docs.spidra.io/sdks/ruby Official Ruby SDK for the Spidra web scraping API. Zero external dependencies. Scrape, batch-process URLs, and crawl entire websites from Ruby. The Ruby SDK wraps the Spidra API so you're not manually firing HTTP requests and writing polling loops from scratch. It handles job submission, status polling, and error mapping, and it pulls in zero external dependencies. Everything runs on the standard library. ## Installation ```bash theme={null} gem install spidra ``` Or add it to your Gemfile: ```ruby theme={null} gem "spidra" ``` Requires Ruby 2.7 or higher. Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Store it as an environment variable. Never hardcode it. ## Getting started ```ruby theme={null} require "spidra" client = Spidra.new(ENV["SPIDRA_API_KEY"]) ``` From here you access everything through `client.scrape`, `client.batch`, `client.crawl`, `client.logs`, and `client.usage`. If you need to point at a different host or change the HTTP timeout, pass those as keyword arguments: ```ruby theme={null} client = Spidra.new( ENV["SPIDRA_API_KEY"], base_url: "http://localhost:4321/api", timeout: 60 ) ``` ## Scraping The scraper accepts up to three URLs per request and processes them in parallel. You can pass a plain extraction prompt, a full JSON schema, per-URL browser actions, or any mix of those. The simplest path is `run` — it submits the job and blocks until it finishes, then returns the result: ```ruby theme={null} job = client.scrape.run( urls: [{ url: "https://example.com/pricing" }], prompt: "Extract all pricing plans with name, price, and included features", output: "json" ) puts job["result"]["content"] # {"plans" => [{"name" => "Starter", "price" => "$9/mo", ...}]} ``` If you'd rather fire and move on, `submit` returns a job ID immediately and you call `get` whenever you're ready to check: ```ruby theme={null} response = client.scrape.submit( urls: [{ url: "https://example.com" }], prompt: "Extract the main headline" ) job_id = response["jobId"] # Later... status = client.scrape.get(job_id) if status["status"] == "completed" puts status["result"]["content"] end ``` Job statuses move through: `waiting` → `active` → `completed` (or `failed`). ### Scrape parameters | Parameter | Type | Description | | -------------------- | ------- | --------------------------------------------------------------------- | | `urls` | Array | Up to 3 URLs. Each entry is `{ url: "..." }` with optional `actions:` | | `prompt` | String | What to extract, written in plain English | | `output` | String | `"markdown"` (default) or `"json"` | | `schema` | Hash | JSON Schema that forces a specific output shape | | `useProxy` | Boolean | Route through a residential proxy | | `proxyCountry` | String | Two-letter country code: `"us"`, `"de"`, `"jp"`, etc. | | `extractContentOnly` | Boolean | Strip nav, ads, and boilerplate before the AI sees the page | | `screenshot` | Boolean | Capture a viewport screenshot | | `fullPageScreenshot` | Boolean | Capture a full-page scrolled screenshot | | `cookies` | String | Raw `Cookie` header string for pages behind a login | ### Enforcing an exact output shape Without a schema the AI extracts what it finds. With a schema, missing fields come back as `null` rather than guessed values, which matters when the output feeds a database or a typed pipeline downstream: ```ruby theme={null} job = client.scrape.run( urls: [{ url: "https://jobs.example.com/senior-engineer" }], prompt: "Extract the job listing details", output: "json", schema: { type: "object", required: ["title", "company", "remote"], properties: { title: { type: "string" }, company: { type: "string" }, remote: { type: ["boolean", "null"] }, salary_min: { type: ["number", "null"] }, skills: { type: "array", items: { type: "string" } } } } ) ``` ### Scraping geo-restricted content Some sites serve different prices or content depending on where you're browsing from. Set `useProxy` and `proxyCountry` to route through a residential IP in that country: ```ruby theme={null} job = client.scrape.run( urls: [{ url: "https://www.amazon.de/gp/bestsellers" }], prompt: "List the top 10 products with name and price", useProxy: true, proxyCountry: "de" ) ``` Supported country codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing without pinning to a specific country. ### Scraping pages behind a login If the page requires a session, pass your cookies as a raw header string. The easiest way to get this is to log in through your browser, open devtools, and copy the `Cookie` header from any authenticated request: ```ruby theme={null} job = client.scrape.run( urls: [{ url: "https://app.example.com/dashboard" }], prompt: "Extract the monthly revenue and active user count", cookies: "session=abc123; auth_token=xyz789" ) ``` ### Browser actions Sometimes you need to interact with the page before extraction — dismiss a cookie banner, type into a search box, scroll to load lazy content. Pass an `actions` array inside the URL entry and they run in order before the AI sees the page: ```ruby theme={null} job = client.scrape.run( urls: [ { url: "https://example.com/products", actions: [ { type: "click", selector: "#accept-cookies" }, { type: "wait", duration: 1000 }, { type: "scroll", to: "80%" } ] } ], prompt: "Extract all product names and prices visible on the page" ) ``` For `selector` you can pass a CSS selector or XPath. If you'd rather describe the element in plain English, use `value` and Spidra will locate it with AI. | Action | What it does | | --------- | ------------------------------------------------------------------ | | `click` | Click any element — use `selector` for CSS, `value` for plain text | | `type` | Type into an input or textarea | | `check` | Check a checkbox | | `uncheck` | Uncheck a checkbox | | `wait` | Pause for `duration` milliseconds | | `scroll` | Scroll to a percentage of the page height, e.g. `"80%"` | | `forEach` | Loop over every matched element and extract from each one | ### Controlling how long run waits By default `run` polls every 3 seconds and gives up after 120 seconds. You can override both by passing keyword arguments after the params hash: ```ruby theme={null} job = client.scrape.run( { urls: [{ url: "https://example.com" }], prompt: "..." }, poll_interval: 5, # seconds between checks timeout: 60 # give up after this many seconds ) ``` The same options work on `batch.run` and `crawl.run`. ## Batch scraping When you have a list of URLs to process, batch is the right tool. You can submit up to 50 URLs in a single request and they all run in parallel. Unlike the scraper, each URL here is a plain string — there's no per-URL actions support in batch mode. ```ruby theme={null} batch = client.batch.run( urls: [ "https://shop.example.com/product/1", "https://shop.example.com/product/2", "https://shop.example.com/product/3" ], prompt: "Extract product name, price, and whether it is in stock", output: "json" ) puts "#{batch["completedCount"]}/#{batch["totalUrls"]} completed" batch["items"].each do |item| if item["status"] == "completed" puts item["result"].inspect else puts "Failed: #{item["url"]} — #{item["error"]}" end end ``` Each item moves through `pending` → `running` → `completed` (or `failed`). The batch itself follows the same lifecycle, plus a `cancelled` state if you stop it early. If you don't want to wait for the whole thing to finish, use `submit` and `get` separately: ```ruby theme={null} response = client.batch.submit( urls: ["https://example.com/1", "https://example.com/2"], prompt: "Extract the page title and meta description" ) batch_id = response["batchId"] # Come back later result = client.batch.get(batch_id) puts "#{result["completedCount"]} of #{result["totalUrls"]} done" ``` ### Retrying failures and cancelling If some items fail due to timeouts or transient errors, you can retry just those without re-running the ones that already succeeded: ```ruby theme={null} if batch["failedCount"] > 0 client.batch.retry(batch_id) end ``` To stop a running batch and get credits back for anything that hasn't started yet: ```ruby theme={null} client.batch.cancel(batch_id) ``` To look through past batches: ```ruby theme={null} page = client.batch.list(1, 20) # page, limit page["jobs"].each do |job| puts "#{job["uuid"]} #{job["status"]} — #{job["completedCount"]}/#{job["totalUrls"]}" end ``` ## Crawling Crawling is different from scraping. You give it a starting URL and it discovers and processes pages on its own, following links according to your instructions. Good for indexing a docs site, monitoring a competitor's blog, or building a structured dataset from an entire section of a site. ```ruby theme={null} job = client.crawl.run( { baseUrl: "https://competitor.com/blog", crawlInstruction: "Follow links to blog posts only, skip tag pages, category pages, and the homepage", transformInstruction: "Extract the post title, author name, publish date, and a one-sentence summary", maxPages: 30, useProxy: true }, timeout: 360 ) job["result"].each do |page| puts "#{page["url"]}: #{page["data"].inspect}" end ``` `crawlInstruction` tells the crawler which links to follow. `transformInstruction` tells the AI what to extract from each page it visits. `maxPages` is a safety cap so the crawl doesn't run indefinitely. The default timeout for `crawl.run` is 300 seconds — pass a higher value for bigger crawls. The same `useProxy`, `proxyCountry`, and `cookies` options from the scraper work here too. Just like scraping, you can fire-and-forget with `submit` and poll with `get`: ```ruby theme={null} response = client.crawl.submit( baseUrl: "https://example.com/docs", crawlInstruction: "Follow all documentation pages", transformInstruction: "Extract the page title and a short summary of the content", maxPages: 50 ) job_id = response["jobId"] status = client.crawl.get(job_id) # status moves through: waiting → active → running → completed (or failed) ``` ### Downloading the raw content Once a crawl completes, you can fetch signed URLs to download the raw HTML and Markdown for every page that was crawled. These links expire after an hour: ```ruby theme={null} result = client.crawl.pages(job_id) result["pages"].each do |page| # page["html_url"] — download the raw HTML # page["markdown_url"] — download the cleaned Markdown puts "#{page["url"]} — #{page["status"]}" end ``` ### Re-extracting with a different prompt If you crawled a site and want to pull out different information, you don't have to re-crawl. `extract` runs a new AI pass over the already-crawled content and charges only transformation credits: ```ruby theme={null} result = client.crawl.extract( completed_job_id, "Extract only product SKUs and prices as structured JSON" ) # This creates a new job — poll it like any other extracted = client.crawl.get(result["jobId"]) ``` ### Browsing your crawl history ```ruby theme={null} history = client.crawl.history(1, 10) puts "Total crawl jobs: #{history["total"]}" stats = client.crawl.stats puts "All-time: #{stats["total"]}" ``` ## Logs Every scrape request your API key makes gets logged automatically. You can filter by status, URL, date range, or where it came from: ```ruby theme={null} result = client.logs.list( status: "failed", searchTerm: "amazon.com", dateStart: "2024-01-01", dateEnd: "2024-12-31", page: 1, limit: 20 ) result.dig("data", "logs").each do |log| puts "#{log["urls"][0]["url"]} — #{log["status"]} (#{log["credits_used"]} credits)" end ``` To fetch the full details of a single log entry including the AI extraction output: ```ruby theme={null} log = client.logs.get(log_uuid) puts log.inspect ``` ## Usage statistics Check how many requests and credits your account has used over a given period: ```ruby theme={null} result = client.usage.get("30d") # "7d" | "30d" | "weekly" result["data"].each do |row| puts "#{row["date"]}: #{row["requests"]} requests, #{row["credits"]} credits" end ``` `"7d"` gives one row per day for the last week. `"30d"` gives the last month. `"weekly"` gives one row per week for the last seven weeks. ## Error handling Every API error is mapped to a typed exception class, so you can rescue exactly what you care about and let the rest bubble up: ```ruby theme={null} begin job = client.scrape.run( urls: [{ url: "https://example.com" }], prompt: "Extract the main headline" ) rescue Spidra::AuthenticationError # Bad or missing API key rescue Spidra::InsufficientCreditsError # Account is out of credits — time to top up rescue Spidra::RateLimitError # Slow down — you're hitting limits rescue Spidra::ServerError => e # Something went wrong on Spidra's side — retry is usually safe puts "Server error (#{e.status}): #{e.message}" rescue Spidra::Error => e # Catch-all for anything else puts "API error #{e.status}: #{e.message}" end ``` | Exception | Status | When | | ---------------------------------- | ------ | ------------------------------------ | | `Spidra::AuthenticationError` | 401 | The API key is missing or invalid | | `Spidra::InsufficientCreditsError` | 403 | No credits remaining on the account | | `Spidra::RateLimitError` | 429 | Too many requests — back off | | `Spidra::ServerError` | 500 | Unexpected server-side error | | `Spidra::Error` | any | Base class for all Spidra exceptions | All exceptions expose `.status` for the HTTP status code and `.message` for a human-readable explanation. Official Go SDK — typed structs, idiomatic errors, zero external dependencies. Official Python SDK — async-first with sync wrappers. Works in scripts, Django, Flask, and Jupyter. # Rust Source: https://docs.spidra.io/sdks/rust Official Rust SDK for Spidra — AI-powered web scraping with proxy rotation and CAPTCHA handling. The official Rust SDK for Spidra lets you extract structured data from any website by describing what you want in plain English. It handles JavaScript rendering, anti-bot bypass, and CAPTCHA solving as a managed API, so your Rust code stays focused on the data. Built on `tokio` and `reqwest`, the SDK uses native Rust async/await throughout and returns `Result` on every call. ## Installation Add the dependency to your `Cargo.toml`: ```toml theme={null} [dependencies] spidra = "0.1" ``` Or via Cargo: ```sh theme={null} cargo add spidra ``` Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Keep your key out of source control — read it from an environment variable. *** ## Getting started All requests require an API key. Pass it to the client: ```rust theme={null} let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").expect("SPIDRA_API_KEY not set")); ``` ## Quick start ```rust theme={null} use spidra::{SpidraClient, types::ScrapeParams}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new(std::env::var("SPIDRA_API_KEY").unwrap()); let result = client .scrape() .run(&ScrapeParams::new("https://example.com")) .await?; println!("{}", result.content.unwrap_or_default()); Ok(()) } ``` `run()` submits the job and polls until it completes, then returns the result. *** ## Scraping ### Scrape a single page ```rust theme={null} use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); let mut params = ScrapeParams::new("https://news.ycombinator.com"); params.output_format = Some(OutputFormat::Markdown); params.render_js = Some(true); // run() polls until complete; submit() returns immediately with a job ID. let result = client.scrape().run(¶ms).await?; if let Some(content) = result.content { println!("{content}"); } Ok(()) } ``` ### AI extraction with a prompt ```rust theme={null} use spidra::{SpidraClient, types::{ScrapeParams, OutputFormat}}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); let mut params = ScrapeParams::new("https://news.ycombinator.com"); params.output_format = Some(OutputFormat::Json); params.prompt = Some("Extract the top 5 story titles and their URLs".to_string()); let result = client.scrape().run(¶ms).await?; if let Some(data) = result.data { println!("{}", serde_json::to_string_pretty(&data).unwrap()); } Ok(()) } ``` ### Submit and poll manually If you need more control over polling (e.g. to show progress), submit and poll separately: ```rust theme={null} use spidra::{SpidraClient, types::ScrapeParams}; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); // submit() returns immediately with a job ID let queued = client .scrape() .submit(&ScrapeParams::new("https://example.com")) .await?; println!("Job submitted: {}", queued.job_id); // Poll manually loop { let status = client.scrape().get(&queued.job_id).await?; println!("Status: {}", status.status); match status.status.as_str() { "completed" => { if let Some(result) = status.result { println!("{}", result.content.unwrap_or_default()); } break; } "failed" | "cancelled" => { eprintln!("Job ended with status: {}", status.status); break; } _ => tokio::time::sleep(Duration::from_secs(3)).await, } } Ok(()) } ``` **Job statuses:** `waiting` · `active` · `completed` · `failed` · `cancelled` ### Custom polling options Use `run_with_options` to override the default poll interval and timeout: ```rust theme={null} use spidra::{SpidraClient, types::ScrapeParams}; use std::time::Duration; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); let params = ScrapeParams::new("https://example.com"); let result = client .scrape() .run_with_options(¶ms, Duration::from_secs(2), Duration::from_secs(60)) .await?; println!("{}", result.content.unwrap_or_default()); Ok(()) } ``` *** ## Batch scraping Submit up to 50 URLs in a single request. All URLs are processed in parallel. ```rust theme={null} use spidra::{SpidraClient, types::{BatchScrapeParams, ScrapeUrl, OutputFormat}}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); let urls = vec![ ScrapeUrl::new("https://example.com"), ScrapeUrl::new("https://example.org"), ScrapeUrl::new("https://example.net"), ]; let params = BatchScrapeParams::new(urls); // run() submits and blocks until the whole batch finishes. let response = client.batch().run(¶ms).await?; println!( "Completed: {} / Failed: {}", response.completed_count, response.failed_count ); for item in &response.items { if let Some(result) = &item.result { println!("{}: {} chars", item.url, result.content.as_deref().unwrap_or("").len()); } } Ok(()) } ``` **Item statuses:** `pending` · `running` · `completed` · `failed` **Batch statuses:** `pending` · `running` · `completed` · `failed` · `cancelled` *** ## Crawling Give Spidra a starting URL and instructions for which links to follow. It discovers pages automatically and extracts structured data from each one. ```rust theme={null} use spidra::{SpidraClient, types::CrawlParams}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); let mut params = CrawlParams::new("https://docs.example.com"); params.max_pages = Some(50); params.max_depth = Some(3); params.include_paths = Some(vec!["docs/**".to_string()]); let pages = client.crawl().run(¶ms).await?; println!("Crawled {} pages", pages.len()); for page in &pages { println!(" {} — {}", page.status, page.url); } Ok(()) } ``` *** ## Logs Every API scrape job is logged automatically. ```rust theme={null} use spidra::{SpidraClient, types::ScrapeLogsParams}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); let mut params = ScrapeLogsParams::default(); params.limit = Some(20); params.status = Some("completed".to_string()); let logs = client.logs().list(¶ms).await?; for log in &logs.logs { println!("{} — {} — {:?} credits", log.uuid, log.url, log.credits_used); } Ok(()) } ``` *** ## Usage statistics Returns credit and request usage broken down by day or week. ```rust theme={null} use spidra::{SpidraClient, types::UsageRange}; #[tokio::main] async fn main() -> Result<(), spidra::SpidraError> { let client = SpidraClient::new("your-api-key"); let rows = client.usage().get(UsageRange::ThirtyDays).await?; for row in &rows { println!("{}: {} requests, {:.2} credits", row.date, row.requests, row.credits); } Ok(()) } ``` | Range | Description | | ------------------------ | ------------------------------ | | `UsageRange::SevenDays` | Last 7 days, one row per day | | `UsageRange::ThirtyDays` | Last 30 days, one row per day | | `UsageRange::Weekly` | Last 7 weeks, one row per week | *** ## Error handling All methods return `Result`. The error variants are: | Variant | When | | ---------------------------------------------- | -------------------------------------- | | `SpidraError::Auth` | 401 / 403 — invalid or missing API key | | `SpidraError::RateLimit` | 429 — rate limit exceeded | | `SpidraError::InsufficientCredits` | 402 — not enough credits | | `SpidraError::Api { status, message }` | Other 4xx errors | | `SpidraError::ServerError { status, message }` | 5xx server errors | | `SpidraError::Timeout { seconds }` | Polling timed out | | `SpidraError::JobFailed { message }` | Remote job reached failed state | | `SpidraError::JobCancelled` | Remote job was cancelled | | `SpidraError::Http(e)` | Underlying `reqwest` transport error | | `SpidraError::Json(e)` | JSON deserialisation error | ```rust theme={null} match client.scrape().run(¶ms).await { Ok(result) => println!("{}", result.content.unwrap_or_default()), Err(spidra::SpidraError::Auth) => eprintln!("Invalid or missing API key"), Err(spidra::SpidraError::RateLimit) => eprintln!("Rate limited — back off and retry"), Err(spidra::SpidraError::InsufficientCredits) => eprintln!("Out of credits"), Err(spidra::SpidraError::Timeout { seconds }) => { eprintln!("Timed out after {}s", seconds) } Err(e) => eprintln!("Error: {e}"), } ``` Official Java SDK — CompletableFuture-based, builder pattern, no extra HTTP dependencies. # Swift Source: https://docs.spidra.io/sdks/swift Official Swift SDK for Spidra — scrape pages, run browser actions, batch-process URLs, and crawl entire sites using modern Swift Concurrency. The official Swift SDK for Spidra uses modern `async/await` concurrency throughout. All results come back as structured data ready to feed into your iOS, macOS, or server-side Swift applications. ## Installation ### Swift Package Manager Add Spidra to your `Package.swift` dependencies: ```swift theme={null} dependencies: [ .package(url: "https://github.com/spidra-io/spidra-swift.git", from: "1.0.0") ] ``` Or add it directly via Xcode: **File → Add Packages...** and paste the repository URL. Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**. Never hardcode it in source files — use an environment variable instead. ## Requirements * Swift 5.9+ * iOS 15.0+ / macOS 12.0+ / tvOS 15.0+ / watchOS 8.0+ * A Spidra API key ([sign up free](https://spidra.io)) *** ## Getting started ```swift theme={null} import Spidra let spidra = SpidraClient(apiKey: "spd_YOUR_API_KEY") ``` From here you access everything through `spidra.scrape`, `spidra.batch`, `spidra.crawl`, `spidra.logs`, and `spidra.usage`. ## Quick start ```swift theme={null} import Spidra Task { do { let spidra = SpidraClient(apiKey: "spd_YOUR_API_KEY") let params = ScrapeParams( urls: [ScrapeUrl(url: "https://news.ycombinator.com")], prompt: "List the top 5 stories with title, points, and comment count", output: "json" ) let job = try await spidra.scrape.run(params) if let content = job.result?.content?.value { print(content) } } catch { print("Error: \(error.localizedDescription)") } } ``` *** ## Scraping All scrape jobs run asynchronously using Swift's `async/await`. The `run()` method submits a job and polls until it finishes. Up to 3 URLs can be passed per request and they are processed in parallel. ### Basic scrape ```swift theme={null} let params = ScrapeParams( urls: [ScrapeUrl(url: "https://example.com/pricing")], prompt: "Extract all pricing plans with name, price, and included features", output: "json" ) let job = try await spidra.scrape.run(params) print(job.result?.content?.value ?? "No data") ``` **Parameters** | Parameter | Type | Description | | -------------------- | ------------- | ----------------------------------------------------------- | | `urls` | `[ScrapeUrl]` | Up to 3 URLs, each with optional per-URL browser actions | | `prompt` | `String` | AI extraction instruction | | `output` | `String` | `"markdown"` (default) or `"json"` | | `schema` | `AnyCodable?` | JSON Schema for guaranteed output shape | | `useProxy` | `Bool` | Route through a residential proxy | | `proxyCountry` | `String?` | Two-letter country code, e.g. `"us"`, `"de"`, `"jp"` | | `extractContentOnly` | `Bool` | Strip navigation, ads, and boilerplate before AI extraction | | `screenshot` | `Bool` | Capture a screenshot of the page | | `fullPageScreenshot` | `Bool` | Capture a full-page (scrolled) screenshot | | `cookies` | `String?` | Raw `Cookie` header string for authenticated pages | ### Fire-and-forget approach Use `submit()` and `get()` when you want to manage polling yourself. ```swift theme={null} // Submit a job immediately let queued = try await spidra.scrape.submit(ScrapeParams( urls: [ScrapeUrl(url: "https://example.com")], prompt: "Extract the main headline" )) // Check status later let status = try await spidra.scrape.get(queued.jobId) if status.status == "completed" { print(status.result?.content?.value ?? "") } ``` **Job statuses:** `waiting` · `active` · `completed` · `failed` ### Structured JSON output Pass a `schema` to enforce an exact output shape. Missing fields come back as `null` rather than hallucinated values. ```swift theme={null} let schemaDict: [String: Any] = [ "type": "object", "required": ["title", "company", "remote"], "properties": [ "title": ["type": "string"], "company": ["type": "string"], "remote": ["type": ["boolean", "null"]] ] ] let params = ScrapeParams( urls: [ScrapeUrl(url: "https://jobs.example.com/senior-engineer")], prompt: "Extract the job listing details", output: "json", schema: AnyCodable(schemaDict) ) let job = try await spidra.scrape.run(params) ``` ### Geo-targeted scraping Pass `useProxy: true` and a `proxyCountry` code to route through a residential IP in that country. ```swift theme={null} let params = ScrapeParams( urls: [ScrapeUrl(url: "https://www.amazon.de/gp/bestsellers")], prompt: "List the top 10 products with name and price", useProxy: true, proxyCountry: "de" ) ``` Supported codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing. ### Authenticated pages Pass cookies as a string to scrape pages that require a login session. ```swift theme={null} let params = ScrapeParams( urls: [ScrapeUrl(url: "https://app.example.com/dashboard")], prompt: "Extract the monthly revenue and active user count", cookies: "session=abc123; auth_token=xyz789" ) ``` ### Browser actions Actions let you interact with the page before the scrape runs. They execute in order. ```swift theme={null} let url = ScrapeUrl( url: "https://example.com/products", actions: [ .click(selector: "#accept-cookies", value: nil), .wait(duration: 1000), .scroll(to: "80%") ] ) let params = ScrapeParams(urls: [url], prompt: "Extract product names and prices") let job = try await spidra.scrape.run(params) ``` **Available actions** | Action | Description | | ---------------------------- | ------------------------------------------------ | | `.click(selector:value:)` | Click a button, link, or any element | | `.type(selector:value:)` | Type text into an input or textarea | | `.check(selector:value:)` | Check a checkbox | | `.uncheck(selector:value:)` | Uncheck a checkbox | | `.wait(duration:)` | Pause for a set number of milliseconds | | `.scroll(to:)` | Scroll to a percentage of the page height | | `.forEach(observe:mode:...)` | Loop over every matched element and process each | ### forEach — loop over every element `forEach` finds a set of elements and processes each individually. Best used when dealing with pagination, clicking into detail pages, or looping over long lists. ```swift theme={null} let forEachAction = BrowserAction.forEach( observe: "Find all book cards in the product grid", mode: "inline", captureSelector: "article.product_pod", maxItems: 20, itemPrompt: "Extract title, price, and star rating. Return as JSON", waitAfterClick: nil, actions: nil, pagination: nil ) let url = ScrapeUrl( url: "https://books.toscrape.com/", actions: [forEachAction] ) ``` **Modes:** * **`inline`** — Read element content directly without navigating away. * **`navigate`** — Follow each element's link to its destination page and capture content there. * **`click`** — Click each element, capture the content that appears (e.g., a modal), then move on. You can also use pagination to navigate through multiple pages automatically: ```swift theme={null} let pagination = BrowserActionPagination(nextSelector: "li.next > a", maxPages: 3) ``` ### Poll options Override default polling intervals via `PollOptions`: ```swift theme={null} let options = PollOptions(pollInterval: 2.0, timeout: 60.0) let job = try await spidra.scrape.run(params, options: options) ``` The same options work on `batch.run()` and `crawl.run()`. *** ## Batch scraping Submit up to 50 URLs in a single request. All URLs are processed in parallel. Each URL is a plain string. ```swift theme={null} let params = BatchScrapeParams( urls: [ "https://shop.example.com/product/1", "https://shop.example.com/product/2", "https://shop.example.com/product/3" ], prompt: "Extract product name, price, and availability", output: "json", useProxy: true ) let batch = try await spidra.batch.run(params) for item in batch.items { if item.status == "completed" { print("Completed: \(item.url)") } else if item.status == "failed" { print("Failed: \(item.error ?? "Unknown")") } } ``` **Item statuses:** `pending` · `running` · `completed` · `failed` **Batch statuses:** `pending` · `running` · `completed` · `failed` · `cancelled` You can also `list()`, `retry()`, or `cancel()` batches using the same pattern as `scrape`. *** ## Crawling Given a starting URL, Spidra discovers pages automatically according to your instruction and extracts structured data from each one. ```swift theme={null} let params = CrawlParams( baseUrl: "https://competitor.com/blog", crawlInstruction: "Find all blog posts published in 2024", transformInstruction: "Extract the title, author, and publish date", maxPages: 30 ) let job = try await spidra.crawl.run(params) if let pages = job.result { for page in pages { print(page.url, page.data?.value ?? "No Data") } } ``` **Parameters** | Parameter | Type | Description | | ---------------------- | --------- | -------------------------------------------------- | | `baseUrl` | `String` | Starting URL for the crawl | | `crawlInstruction` | `String` | Which links to follow and which to skip | | `transformInstruction` | `String` | What to extract from each page | | `maxPages` | `Int` | Maximum number of pages to crawl | | `useProxy` | `Bool` | Route through a residential proxy | | `proxyCountry` | `String?` | Two-letter country code, e.g. `"us"` | | `cookies` | `String?` | Raw `Cookie` header string for authenticated sites | ### Download crawled content Fetch signed download URLs for HTML and Markdown for all crawled pages. Links expire after **1 hour**. ```swift theme={null} let response = try await spidra.crawl.pages(job.jobId) ``` *** ## Logs Every API scrape job is logged automatically. ```swift theme={null} let params = ScrapeLogsParams(status: "failed", limit: 20) let response = try await spidra.logs.list(params) for log in response.logs { print("Log: \(log.uuid) - Status: \(log.status) - Credits: \(log.creditsUsed)") } // Get full extraction result for a specific log let detail = try await spidra.logs.get("log-uuid") ``` *** ## Usage statistics Returns credit and request usage broken down by day or week. ```swift theme={null} let rows = try await spidra.usage.get("30d") // "7d" | "30d" | "weekly" for row in rows { print("Date: \(row.date) - Requests: \(row.requests) - Credits: \(row.credits)") } ``` | Range | Description | | ---------- | ------------------------------ | | `"7d"` | Last 7 days, one row per day | | `"30d"` | Last 30 days, one row per day | | `"weekly"` | Last 7 weeks, one row per week | *** ## Error handling Every API error throws a `SpidraError`. Catch the specific case you care about. ```swift theme={null} do { let job = try await spidra.scrape.run(params) } catch SpidraError.authenticationError(let msg) { // 401: API key is missing or invalid print("Check your API key: \(msg)") } catch SpidraError.insufficientCreditsError(let msg) { // 403: Monthly credit limit reached print("Out of credits: \(msg)") } catch SpidraError.rateLimitError(let msg) { // 429: Too many requests print("Rate limited: \(msg)") } catch SpidraError.serverError(let msg) { // 500: Server error print("Server error: \(msg)") } catch { // Decoding errors, network timeouts, etc. print("Other error: \(error.localizedDescription)") } ``` | Error case | Status | When | | --------------------------- | ------ | ----------------------------- | | `.authenticationError` | 401 | API key is missing or invalid | | `.insufficientCreditsError` | 403 | No credits remaining | | `.rateLimitError` | 429 | Too many requests — back off | | `.serverError` | 500 | Unexpected server-side error | Official .NET SDK — fully async, typed exceptions, JSON schema support. Requires .NET 8+. Official Java SDK — CompletableFuture-based, builder pattern, no extra HTTP dependencies.