Skip to main content

Base URL

All requests use HTTPS. Request bodies are JSON (Content-Type: application/json), except document uploads to Parse, which are multipart/form-data.

OpenAPI spec

The whole API is described in one OpenAPI 3 file, ready for Postman, Insomnia, client generators and AI coding assistants:
That address redirects to the file itself, https://docs.spidra.io/api-reference/openapi.json.

Authentication

Send your API key in the Authorization header. x-api-key: <key> also works. See Authentication.

Jobs and polling

Endpoints that read pages run as jobs: Submit returns 202 with the ID. Poll every 2 to 5 seconds until the status is completed, failed or cancelled. Polling is free and isn’t rate limited. These answer in the same request instead:
  • POST /map always.
  • POST /search with "wait": true (up to 20 seconds).
  • POST /batch/scrape with "wait": true and up to 10 URLs (up to 45 seconds).
  • POST /parse by default (up to 20 seconds; "wait": false to opt out).
  • POST /templates/{slug}/runs with "wait": true.
A request that can’t finish in time answers 202 with an ID, as above.

Conventions

  • Field names are camelCase in requests. Most responses are camelCase too; a few older ones (scrape results, scrape and search logs, crawl job details) also use snake_case fields such as ai_extraction_failed and credits_used.
  • IDs are UUIDs.
  • Times are ISO 8601 in UTC.
  • Lists page with page and limit (most list endpoints) or offset and limit (crawl pages, crawl report). Responses include a total.
  • Credits used by a request are in its response or its job (creditsUsed, credits_used).
  • URLs must be public http or https addresses. Private and internal addresses are refused.

Responses and errors

Errors look like this:
Some add a code and, for validation problems, an errors list. Errors has every status and code, and Rate limits has the limits per endpoint.

HTTP clients

curl, fetch, axios, requests, httpx and the SDKs all work. Python’s built-in urllib and Perl’s libwww-perl send default user agents that are blocked with a 403 (error code: 1010); use another client or set your own User-Agent.

SDKs and the MCP server

The SDKs wrap this API in ten languages, poll for you and retry when Spidra is briefly unavailable. The MCP server exposes it to AI assistants.