Skip to main content
POST
Submit a Batch Scrape Job

How It Works

Batch scrape jobs are asynchronous. Submitting returns a batchId immediately. Each URL is processed in parallel by independent workers.
  1. Submit — Send your URL list. Receive batchId in the response.
  2. Process — Each URL is opened in a real browser, CAPTCHAs solved, content extracted.
  3. Poll — Call GET /api/batch/scrape/{batchId} every 2–5 seconds until status is terminal.
Credits are reserved upfront when you submit. The final amount is reconciled per item once processing completes.

Minimal Example

Response 202 Accepted:
The Location response header is also set to /api/batch/scrape/{batchId} for convenience.

With Structured Output

Pass a schema to receive a consistent JSON shape for every item:
When schema is provided, output is automatically forced to "json". Non-fatal schema issues are returned as schema_warnings in the submission response.
Use the Spidra JSON Schema Generator to build and preview your schema visually before pasting it here.

With Proxy


With Screenshots

Screenshot URLs are returned in each item’s screenshotUrl field once processing is complete.

Request Body


Response


Errors

Validation error example:

Get Batch Status

Poll for results

Batch Scraping Guide

Full feature walkthrough

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json
urls
string<uri>[]
required

URLs to scrape. 1–50 per request. Must be http:// or https://. Private/internal IPs are rejected.

Required array length: 1 - 50 elements
prompt
string

AI extraction instruction applied to every URL in the batch

output
enum<string>
default:json

Output format. Automatically set to 'json' when schema is provided

Available options:
json,
markdown
schema
object

JSON Schema that constrains AI output shape. Validated before queuing — returns 422 if invalid

useProxy
boolean
default:false

Route each URL through residential stealth proxies. Usage is billed from your bandwidth quota.

proxyCountry
string

ISO country code (e.g. 'us', 'de', 'gb') or region ('eu', 'global'). Requires useProxy: true

crawlerMode
string
default:default

Browser rendering mode: 'default', 'fast', or 'ai'

extractContentOnly
boolean
default:false

Strip navigation, headers, and sidebars — keep only the main content

cookies
string

Session cookies for authenticated pages. Never persisted — passed ephemerally to the worker

screenshot
boolean
default:false

Capture a viewport screenshot of each page

fullPageScreenshot
boolean
default:false

Capture the full scrollable page. Requires screenshot: true

Response

Batch accepted and queued

status
enum<string>
Available options:
queued
batchId
string<uuid>
total
integer
schema_warnings
string[]

Non-fatal schema issues. Only present if there are warnings