Submit a Scrape Job

curl --request POST \
  --url https://api.spidra.io/api/scrape \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "urls": [
    {
      "url": "https://example.com"
    }
  ],
  "prompt": "Extract the main heading and first paragraph",
  "output": "json"
}
'

{
  "status": "queued",
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Scrape job has been queued. Poll /api/scrape/550e8400-e29b-41d4-a716-446655440000 to get the result."
}

{
  "status": "error",
  "message": "You must pass 1 to 3 URLs with actions."
}

{
  "status": "error",
  "message": "Monthly scrape limit exceeded for your plan."
}

POST

scrape

curl --request POST \
  --url https://api.spidra.io/api/scrape \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "urls": [
    {
      "url": "https://example.com"
    }
  ],
  "prompt": "Extract the main heading and first paragraph",
  "output": "json"
}
'

{
  "status": "queued",
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Scrape job has been queued. Poll /api/scrape/550e8400-e29b-41d4-a716-446655440000 to get the result."
}

{
  "status": "error",
  "message": "You must pass 1 to 3 URLs with actions."
}

{
  "status": "error",
  "message": "Monthly scrape limit exceeded for your plan."
}

How It Works

Spidra runs scrape jobs asynchronously. When you submit a request, you get a jobId back immediately. You then poll GET /scrape/{jobId} until status is completed and results are ready.

Submit - Send your request, receive a jobId in the response right away
Load - Spidra opens each URL in a real browser
Execute - Runs your browser actions (clicks, scrolls, etc.)
Solve - Automatically handles CAPTCHAs
Process - Runs AI extraction if a prompt is provided
Poll - Check GET /scrape/{jobId} until status: "completed"

A successful scrape costs 1 credit per URL, plus a few more if AI extraction runs. See How credits work for the full breakdown, and check your balance any time with GET /account.

Setting output: "json" without a prompt still triggers a default AI extraction pass. If you want raw markdown with no AI processing, omit both output and prompt.When AI extraction fails (for example, on a near-empty page), Spidra falls back to returning the raw page markdown in result.content. Check the ai_extraction_failed flag in the response to detect this case and handle degraded results in your code.

Structured Output

Pass a schema to tell the AI exactly what shape to return. Instead of getting whatever JSON the AI decides to produce, you get back a JSON object that matches your schema every time. Nullable fields come back as null rather than being omitted. Field names match exactly what you defined.

{
  "urls": [{ "url": "https://jobs.example.com/engineer" }],
  "prompt": "Extract the job details. Normalize salary to a plain number in USD.",
  "schema": {
    "type": "object",
    "required": ["title", "company", "remote", "employment_type"],
    "properties": {
      "title":           { "type": "string" },
      "company":         { "type": "string" },
      "remote":          { "type": ["boolean", "null"] },
      "salary_min":      { "type": ["number", "null"] },
      "salary_max":      { "type": ["number", "null"] },
      "employment_type": {
        "type": ["string", "null"],
        "enum": ["full_time", "part_time", "contract", null]
      }
    }
  }
}

output is automatically set to "json" when a schema is provided. The schema is validated before the job is queued and a 422 is returned with descriptive errors if the schema is malformed. Non-fatal issues (unsupported keywords) are returned as schema_warnings in the job status response.

Structured Output Guide

Full guide: nested objects, arrays, nullable fields, the required rule, and schema limits

Zod and Pydantic

Generate your schema from an existing Zod or Pydantic model

Schema Generator

Build and preview your schema visually — no JSON knowledge required

Browser Actions

Interact with the page before scraping — click buttons, fill forms, scroll to load content, dismiss modals, and iterate over lists of elements.

{
  "urls": [{
    "url": "https://example.com/products",
    "actions": [
      {"type": "click", "selector": "#accept-cookies"},
      {"type": "wait", "duration": 1500},
      {"type": "scroll", "to": "80%"}
    ]
  }],
  "prompt": "List all product names and prices",
  "output": "json"
}

Available Actions

Action	What it does	Quick example
`click`	Clicks any element on the page, buttons, links, tabs, toggles	`{"type": "click", "selector": "#load-more"}`
`type`	Types text into an input field or search box	`{"type": "type", "selector": "#search", "value": "laptops"}`
`check`	Checks a checkbox	`{"type": "check", "selector": "#in-stock-only"}`
`uncheck`	Unchecks a checkbox	`{"type": "uncheck", "selector": "#newsletter"}`
`wait`	Pauses for a number of milliseconds	`{"type": "wait", "duration": 2000}`
`scroll`	Scrolls the page to a percentage of its height	`{"type": "scroll", "to": "80%"}`
`forEach`	Finds all matching elements and processes each one individually. Supports navigate, click, and inline modes.	`{"type": "forEach", "observe": "Find all product cards", "mode": "navigate"}`

For click, check, and uncheck you can target elements by CSS selector like "selector": "#id" or by a plain English description like "value": "Accept cookies button". Both work.

forEach is the most powerful action. It finds a set of repeating elements (product cards, links, accordion rows) and runs a mini-scrape on each one. It supports three modes (click, inline, navigate), automatic pagination, per-item AI extraction, and per-element sub-actions.

Full Browser Actions Guide

Detailed explanations for every action with real examples, all forEach options, chaining patterns, and sample responses

Proxy and Geo-Targeting

Route requests through residential proxies to avoid detection or access geo-restricted content. Set "useProxy": true and optionally add "proxyCountry" to target a specific location.

{
  "urls": [{"url": "https://amazon.de/dp/B123456"}],
  "prompt": "Extract the product price in euros",
  "output": "json",
  "useProxy": true,
  "proxyCountry": "de"
}

Stealth Mode & Geo-Targeting Guide

Full guide: country list, EU rotation, examples, and credit costs

Extract Content Only

Remove navigation, headers, footers, and sidebars before processing. Useful when you only want the main article or product content.

{
  "urls": [{"url": "https://blog.example.com/article"}],
  "prompt": "Summarize this article",
  "output": "json",
  "extractContentOnly": true
}

Document URLs (PDF, Word, Excel, PowerPoint…)

Scrape isn’t limited to HTML pages. You can point it at a PDF, Word document, Excel spreadsheet, PowerPoint deck, or similar, and Spidra detects the file type automatically and converts it straight to clean markdown, tables included, no HTML page to parse:

{
  "urls": [{ "url": "https://example.com/annual-report.pdf" }],
  "output": "markdown"
}

Supported formats: PDF, DOC/DOCX, XLSX, PPT/PPTX, ODT, RTF, EPUB, CSV.

Scanned or image-only PDF pages (no real text layer) are transcribed with AI, so you still get real text back rather than an empty result.

This works the same way whether you fetch the URL directly with /scrape or via Search’s scrapeOptions, including on PDF Search results.

Screenshots

Capture screenshots of scraped pages for debugging or archival.

{
  "urls": [{"url": "https://example.com"}],
  "prompt": "Extract page title",
  "screenshot": true,
  "fullPageScreenshot": true
}

Option	Description
`screenshot: true`	Capture the visible viewport
`fullPageScreenshot: true`	Capture the entire scrollable page (requires `screenshot: true`)

Screenshot URLs are returned in the screenshots array of the response.

Authentication

Scrape protected pages by providing session cookies:

{
  "urls": [{"url": "https://example.com/dashboard"}],
  "prompt": "Extract account details",
  "output": "json",
  "cookies": "session=eyJ...; auth_token=abc123..."
}

Authenticated Scraping

Full guide on getting cookies and formats

Check Job Status

Poll for results

View Logs

See your scrape history

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

urls

object[]

required

Array of URLs to scrape (1-3 URLs per request)

Required array length: 1 - 3 elements

Show child attributes

prompt

string

Optional LLM prompt for extracting or transforming the scraped content

output

enum<string>

default:json

Output format for the extracted content

Available options:

json,

markdown

useProxy

boolean

default:false

Enable stealth mode with proxy rotation to avoid detection

proxyCountry

string

Country code (e.g., 'us', 'uk', 'de') or region ('global', 'asia', 'eu') for geo-targeted proxy routing. Requires useProxy: true

string

Session cookies for authenticated scraping. Supports standard format (name=value; name2=value2) or raw Chrome DevTools paste format

screenshot

boolean

default:false

Capture a screenshot of each page after scraping

fullPageScreenshot

boolean

default:false

Capture full page screenshot instead of just the viewport. Requires screenshot: true

extractContentOnly

boolean

default:false

Remove headers, footers, navigation, and other non-content elements from the scraped output

schema

object

JSON Schema object describing the exact shape of the AI output. When provided, the AI must return JSON matching this schema. Output is automatically set to 'json'. Root must be type 'object'. Maximum nesting depth: 5. Maximum size: 10KB.

Response

Job successfully queued

status

enum<string>

Available options:

queued

jobId

string

Unique job identifier for polling

message

string

deduplicated

boolean

True if an identical request was made within the last 5 seconds and this returns the existing job ID

Using the API

Scrape Endpoints

Search Endpoints

Crawl Endpoints

Account

Submit a Scrape Job

How It Works

Structured Output

Structured Output Guide

Zod and Pydantic

Schema Generator

Browser Actions

Available Actions

Full Browser Actions Guide

Proxy and Geo-Targeting

Stealth Mode & Geo-Targeting Guide

Extract Content Only

Document URLs (PDF, Word, Excel, PowerPoint…)

Screenshots

Authentication

Authenticated Scraping

Check Job Status

View Logs

Authorizations

Body

Response

​How It Works

​Structured Output

Structured Output Guide

Zod and Pydantic

Schema Generator

​Browser Actions

​Available Actions

Full Browser Actions Guide

​Proxy and Geo-Targeting

Stealth Mode & Geo-Targeting Guide

​Extract Content Only

​Document URLs (PDF, Word, Excel, PowerPoint…)

​Screenshots

​Authentication

Authenticated Scraping

Check Job Status

View Logs

Authorizations

Body

Response

How It Works

Structured Output

Browser Actions

Available Actions

Proxy and Geo-Targeting

Extract Content Only

Document URLs (PDF, Word, Excel, PowerPoint…)

Screenshots

Authentication