Crawl

A crawl starts at one URL, follows links toward the pages you describe, and returns one result per page, up to 10,000 pages depending on your plan. Use it when you know where a set of pages lives but not every URL: a blog, a product category, a docs site, a job board. Already have the URLs? Use Batch scrape. Only want the list of a site’s pages? Map gives it in seconds for 1 credit.

Quick example

Get the title, price and stock of the books in one category:

curl -X POST https://api.spidra.io/api/crawl \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "baseUrl": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
    "crawlInstruction": "Individual book pages",
    "schema": {
      "type": "object",
      "properties": { "title": { "type": "string" }, "price": { "type": "number" }, "in_stock": { "type": "boolean" } }
    },
    "maxPages": 3
  }'

const job = await spidra.crawl({
  baseUrl: "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
  crawlInstruction: "Individual book pages",
  schema: { type: "object", properties: { title: { type: "string" }, price: { type: "number" } } },
  maxPages: 3,
});

for (const page of job.result) console.log(page.url, page.data);
// https://books.toscrape.com/catalogue/sharp-objects_997/index.html { title: 'Sharp Objects', price: 47.82 }

job = client.crawl(
    "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
    crawl_instruction="Individual book pages",
    schema={"type": "object", "properties": {"title": {"type": "string"}, "price": {"type": "number"}}},
    max_pages=3,
)

for page in job.result:
    print(page.url, page.data)
# https://books.toscrape.com/catalogue/sharp-objects_997/index.html {'title': 'Sharp Objects', 'price': 47.82}

The API answers with a jobId; poll GET /crawl/{jobId} until status is completed. The SDKs wait for you. The result has one entry per page (html and markdown links left out):

{
  "status": "completed",
  "result": [
    {
      "url": "https://books.toscrape.com/catalogue/sharp-objects_997/index.html",
      "status": "success",
      "title": "Sharp Objects | Books to Scrape - Sandbox",
      "data": { "title": "Sharp Objects", "price": 47.82, "in_stock": true },
      "depth": 1,
      "parentUrl": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
    }
  ],
  "totalPages": 3
}

All three results are book pages. The category page was read to find them but isn’t returned and wasn’t charged. The crawl cost 5 credits: 3 pages and 2 for the AI.

How a crawl finds pages

It works out what you want. Spidra reads your crawlInstruction and the start page and decides what kind of page you mean. Here it understood “Individual book detail pages in the mystery category.”
It follows links toward those pages, through category, listing and section pages if it has to, and reads the site’s sitemap.
It stays where it started. A crawl from /blog doesn’t wander to /pricing. Set crawlEntireDomain to allow the whole domain.
It returns only the pages you asked for. Pages read only to reach others are navigation pages: not returned, not charged. The start page is returned only if it’s itself the kind of page you asked for.
It extracts. Your transformInstruction or schema runs on every returned page. With neither, each page’s data is its Markdown and no AI runs.

Name the kind of page: Individual job postings works better than jobs, because a jobs site also has search, category and company pages. All pages returns every page in the part of the site you started in. Instructions work in any language.

What you get back

Each page in result has url, status, title, data (your extracted data, or Markdown), depth (clicks from the start page), parentUrl (where it was found), and links to the stored html and markdown. The links expire, so read them soon or download the ZIP. GET /crawl/{jobId} holds the first 100 pages. For bigger crawls it adds "resultTruncated": true, and you read every page with GET /crawl/{jobId}/pages in pages of up to 1,000. The Python SDK’s crawl() reads them all for you. Without a schema, each page’s data takes whatever shape the AI picks, and it can differ between pages. Use a schema when code reads the result.

Why a page wasn’t returned

Every finished crawl has a report: what Spidra looked for, how many navigation pages it read, and every link it skipped with the reason and the page it was on.

curl "https://api.spidra.io/api/crawl/e303fb87-846e-4a30-8aa5-bfca457a5357/report?limit=2" \
  -H "Authorization: Bearer $SPIDRA_API_KEY"

{
  "status": "success",
  "report": {
    "understoodAs": "Individual book detail pages in the mystery category.",
    "navigationPages": 1,
    "skippedTotal": 60,
    "skipped": [{ "reason": "not_relevant", "count": 60, "description": "Didn't match the crawl instruction" }],
    "skippedUrls": [
      { "url": "https://books.toscrape.com/index.html", "reason": "not_relevant", "foundOn": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" }
    ],
    "skippedUrlsTotal": 60
  }
}

Reason	Meaning
`not_relevant`	Didn’t lead to the kind of page you asked for.
`outside_domain`	Another site, or a subdomain while `allowSubdomains` is off.
`outside_start_path`	Same site, outside the part the crawl started in.
`include_pattern`, `exclude_pattern`	Ruled out by your path filters.
`file_type`	An image, video, archive or script, not a page.
`other_version_or_language`	Another version or locale of a page, when you started from a specific one.
`duplicate_page`	The same content under another URL.
`redirected_off_site`	Redirected off the site.
`robots_txt`	Disallowed by robots.txt, with `respectRobots` on.
`action_link`	Add to cart, log out and similar links, never followed.

Missing a page? Make the instruction more specific, add an includePaths pattern that matches it, or raise maxPages.

Options

Option	What it does
`crawlInstruction` (required)	The pages you want, in plain words.
`transformInstruction`	What to extract from each page.
`schema`	A JSON Schema for every page’s `data`.
`maxPages`	Pages to return (default 5). Navigation pages don’t count. Up to 100 on Free, 1,000 on Starter, 5,000 on Builder, 10,000 on Pro.
`maxDepth`	Clicks from the start page. Usually best left unset.
`includePaths`, `excludePaths`	Glob or regex patterns, such as `["/blog/"]` or `["/tag/"]`. A hard boundary.
`crawlEntireDomain`	Follow links anywhere on the domain, not just under the start path.
`allowSubdomains`	Also follow links to subdomains.
`useSitemap`	Also find pages from the sitemap (on by default).
`respectRobots`	Skip pages robots.txt disallows and honour its crawl delay (off by default).
`ignoreQueryParams`	Treat URLs that differ only by query string as one page.
`scrapeMode`	`"fast"` reads pages over HTTP first and opens a browser only when a page needs one.
`useProxy`, `proxyCountry`, `cookies`	As in Scrape.
`webhookUrl`	Get each page pushed to your server. See Webhooks.

Every field is in the API reference.

After the crawl

Extract again without crawling again. POST /crawl/{jobId}/extract runs a new transformInstruction on the saved pages. It’s quick and costs only the AI tokens. Good for trying different prompts.
Retry one page’s extraction with POST /crawl/{jobId}/retry/{pageId} if it came back empty.
Download everything as a ZIP with each page’s data, Markdown and HTML.
Cancel a running crawl with DELETE /crawl/{jobId}. Pages already done are kept.

Cost

1 credit per page returned, plus AI credits when you extract, plus 5 per CAPTCHA solved. Navigation pages, failed pages and failed crawls are free. Re-extracting and page retries cost only AI tokens. When your balance can’t cover maxPages, the crawl isn’t refused: it runs with maxPages lowered to what your credits cover, and the response says so (maxPagesRequested and a message). Credits held for pages that weren’t needed come back at the end.

In the dashboard

Open Crawl in the sidebar: Website, What pages do you want?, What to extract (with Use a schema), and Options for Max pages, Max depth, Stealth Mode, Fast mode and More settings (path filters, sitemap, subdomains, robots.txt, webhook and login cookies). See its pages first opens Map for the same site. Results show as pages finish, and How this crawl worked shows the report above them.

The Crawl page with the form on the left and three book pages in the results on the right

Tips

Start with maxPages of 3 to 5 to check your instruction, then raise it.
Start the crawl at the section you want (/blog, /docs, a category page).
Use the instruction to steer and path filters to fence.
Read the report before raising maxPages when pages are missing.

Map

See a site’s pages first.

Build a RAG knowledge base

Crawl a docs site into clean Markdown.

Structured output

Schemas that keep every page the same.

API reference

Every field and response.

Get started

Tools

Ready-made data

Getting the data you want

Supported sites

Guides

Dashboard

Account and billing

Quick example

How a crawl finds pages

What you get back

Why a page wasn’t returned

Options

After the crawl

Cost

In the dashboard

Tips

Next

Map

Build a RAG knowledge base

Structured output

API reference

​Quick example

​How a crawl finds pages

​What you get back

​Why a page wasn’t returned

​Options

​After the crawl

​Cost

​In the dashboard

​Tips

​Next

Map

Build a RAG knowledge base

Structured output

API reference

Quick example

How a crawl finds pages

What you get back

Why a page wasn’t returned

Options

After the crawl

Cost

In the dashboard

Tips

Next