Introduction

The Spidra API gives you programmatic access to web data. All endpoints share a common base URL, authentication scheme, and response format described on this page.

Features

Scrape

Extract content from any webpage with browser automation and AI processing.

Batch Scrape

Queue up to 50 URLs in one request and collect per-item results in parallel.

Crawl

Crawl entire websites and extract structured data from every page.

Base URL

All API requests use this base URL:

https://api.spidra.io/api

Python’s built-in urllib (its default User-Agent is Python-urllib/3.x) and Perl’s libwww-perl are rejected before they reach Spidra, with a 403 that mentions browser_signature_banned (Cloudflare error 1010). If you see that, use the official SDKs, use requests or httpx, or set your own User-Agent header such as my-app/1.0. curl, fetch, requests, httpx, Go, Java and the other common clients are not affected.

Authentication

Include your API key in the standard Authorization: Bearer header.

curl -X POST https://api.spidra.io/api/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": [{"url": "https://example.com"}]}'

import requests

response = requests.post(
    "https://api.spidra.io/api/scrape",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"urls": [{"url": "https://example.com"}]}
)

const response = await fetch("https://api.spidra.io/api/scrape", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({ urls: [{ url: "https://example.com" }] })
});

Get your API key from app.spidra.io → API Keys.

Keep your API key secret. Never expose it in frontend code.

Response Codes

Code	Meaning	What it indicates
`200`	OK	Request completed successfully
`202`	Accepted	Job queued. Poll the status endpoint for results
`400`	Bad Request	The request conflicts with the current state (e.g. cancelling a finished job)
`401`	Unauthorized	API key is missing, invalid, or expired
`402`	Payment Required	Subscription payment overdue
`403`	Forbidden	Credits exhausted or plan limit reached
`404`	Not Found	The job or resource does not exist
`422`	Unprocessable Entity	The request body failed validation — the response includes an `errors` array
`429`	Too Many Requests	Rate limit hit. Wait and retry with backoff
`500`	Server Error	Something went wrong on our end

Error Format

All errors return this structure:

{
  "status": "error",
  "message": "Detailed error explanation"
}

Using the API

Scrape Endpoints

Search Endpoints

Crawl Endpoints

Account

Features

Scrape

Batch Scrape

Crawl

Base URL

Authentication

Response Codes

Error Format

​Features

Scrape

Batch Scrape

Crawl

​Base URL

​Authentication

​Response Codes

​Error Format

Features

Base URL

Authentication

Response Codes

Error Format