> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl

> Start from one URL, describe the pages you want, and get one result per matching page. Pages read only to find them are free, and a report explains every link skipped.

A crawl starts at one URL, follows links toward the pages you describe, and returns one result per page, up to 10,000 pages depending on your plan. Use it when you know where a set of pages lives but not every URL: a blog, a product category, a docs site, a job board.

Already have the URLs? Use [Batch scrape](https://docs.spidra.io/batch-scrape.md). Only want the list of a site's pages? [Map](https://docs.spidra.io/map.md) gives it in seconds for 1 credit.

## Quick example

Get the title, price and stock of the books in one category:

```bash curl theme={null}
curl -X POST https://api.spidra.io/api/crawl \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "baseUrl": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
    "crawlInstruction": "Individual book pages",
    "schema": {
      "type": "object",
      "properties": { "title": { "type": "string" }, "price": { "type": "number" }, "in_stock": { "type": "boolean" } }
    },
    "maxPages": 3
  }'
```

```typescript Node.js theme={null}
const job = await spidra.crawl({
  baseUrl: "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
  crawlInstruction: "Individual book pages",
  schema: { type: "object", properties: { title: { type: "string" }, price: { type: "number" } } },
  maxPages: 3,
});

for (const page of job.result) console.log(page.url, page.data);
// https://books.toscrape.com/catalogue/sharp-objects_997/index.html { title: 'Sharp Objects', price: 47.82 }
```

```python Python theme={null}
job = client.crawl(
    "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
    crawl_instruction="Individual book pages",
    schema={"type": "object", "properties": {"title": {"type": "string"}, "price": {"type": "number"}}},
    max_pages=3,
)

for page in job.result:
    print(page.url, page.data)
# https://books.toscrape.com/catalogue/sharp-objects_997/index.html {'title': 'Sharp Objects', 'price': 47.82}
```

The API answers with a `jobId`; poll [GET /crawl/\{jobId}](https://docs.spidra.io/api-reference/crawling/crawl-status.md) until `status` is `completed`. The SDKs wait for you. The result has one entry per page (`html` and `markdown` links left out):

```json theme={null}
{
  "status": "completed",
  "result": [
    {
      "url": "https://books.toscrape.com/catalogue/sharp-objects_997/index.html",
      "status": "success",
      "title": "Sharp Objects | Books to Scrape - Sandbox",
      "data": { "title": "Sharp Objects", "price": 47.82, "in_stock": true },
      "depth": 1,
      "parentUrl": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
    }
  ],
  "totalPages": 3
}
```

All three results are book pages. The category page was read to find them but isn't returned and wasn't charged. The crawl cost 5 credits: 3 pages and 2 for the AI.

## How a crawl finds pages

1. **It works out what you want.** Spidra reads your `crawlInstruction` and the start page and decides what kind of page you mean. Here it understood "Individual book detail pages in the mystery category."
2. **It follows links toward those pages,** through category, listing and section pages if it has to, and reads the site's sitemap.
3. **It stays where it started.** A crawl from `/blog` doesn't wander to `/pricing`. Set `crawlEntireDomain` to allow the whole domain.
4. **It returns only the pages you asked for.** Pages read only to reach others are navigation pages: not returned, not charged. The start page is returned only if it's itself the kind of page you asked for.
5. **It extracts.** Your `transformInstruction` or `schema` runs on every returned page. With neither, each page's `data` is its Markdown and no AI runs.

Name the kind of page: `Individual job postings` works better than `jobs`, because a jobs site also has search, category and company pages. `All pages` returns every page in the part of the site you started in. Instructions work in any language.

## What you get back

Each page in `result` has `url`, `status`, `title`, `data` (your extracted data, or Markdown), `depth` (clicks from the start page), `parentUrl` (where it was found), and links to the stored `html` and `markdown`. The links expire, so read them soon or [download the ZIP](https://docs.spidra.io/api-reference/crawling/crawl-download.md).

`GET /crawl/{jobId}` holds the first 100 pages. For bigger crawls it adds `"resultTruncated": true`, and you read every page with [GET /crawl/\{jobId}/pages](https://docs.spidra.io/api-reference/crawling/crawl-pages.md) in pages of up to 1,000. The Python SDK's `crawl()` reads them all for you.

Without a schema, each page's `data` takes whatever shape the AI picks, and it can differ between pages. Use a [schema](https://docs.spidra.io/options/structured-output.md) when code reads the result.

## Why a page wasn't returned

Every finished crawl has a report: what Spidra looked for, how many navigation pages it read, and every link it skipped with the reason and the page it was on.

```bash theme={null}
curl "https://api.spidra.io/api/crawl/e303fb87-846e-4a30-8aa5-bfca457a5357/report?limit=2" \
  -H "Authorization: Bearer $SPIDRA_API_KEY"
```

```json theme={null}
{
  "status": "success",
  "report": {
    "understoodAs": "Individual book detail pages in the mystery category.",
    "navigationPages": 1,
    "skippedTotal": 60,
    "skipped": [{ "reason": "not_relevant", "count": 60, "description": "Didn't match the crawl instruction" }],
    "skippedUrls": [
      { "url": "https://books.toscrape.com/index.html", "reason": "not_relevant", "foundOn": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" }
    ],
    "skippedUrlsTotal": 60
  }
}
```

| Reason | Meaning |
| - | - |
| `not_relevant` | Didn't lead to the kind of page you asked for. |
| `outside_domain` | Another site, or a subdomain while `allowSubdomains` is off. |
| `outside_start_path` | Same site, outside the part the crawl started in. |
| `include_pattern`, `exclude_pattern` | Ruled out by your path filters. |
| `file_type` | An image, video, archive or script, not a page. |
| `other_version_or_language` | Another version or locale of a page, when you started from a specific one. |
| `duplicate_page` | The same content under another URL. |
| `redirected_off_site` | Redirected off the site. |
| `robots_txt` | Disallowed by robots.txt, with `respectRobots` on. |
| `action_link` | Add to cart, log out and similar links, never followed. |

Missing a page? Make the instruction more specific, add an `includePaths` pattern that matches it, or raise `maxPages`.

## Options

| Option | What it does |
| - | - |
| `crawlInstruction` (required) | The pages you want, in plain words. |
| `transformInstruction` | What to extract from each page. |
| `schema` | A JSON Schema for every page's `data`. |
| `maxPages` | Pages to return (default 5). Navigation pages don't count. Up to 100 on Free, 1,000 on Starter, 5,000 on Builder, 10,000 on Pro. |
| `maxDepth` | Clicks from the start page. Usually best left unset. |
| `includePaths`, `excludePaths` | Glob or regex patterns, such as `["/blog/*"]` or `["/tag/*"]`. A hard boundary. |
| `crawlEntireDomain` | Follow links anywhere on the domain, not just under the start path. |
| `allowSubdomains` | Also follow links to subdomains. |
| `useSitemap` | Also find pages from the sitemap (on by default). |
| `respectRobots` | Skip pages robots.txt disallows and honour its crawl delay (off by default). |
| `ignoreQueryParams` | Treat URLs that differ only by query string as one page. |
| `scrapeMode` | `"fast"` reads pages over HTTP first and opens a browser only when a page needs one. |
| `useProxy`, `proxyCountry`, `cookies` | As in [Scrape](https://docs.spidra.io/scrape.md). |
| `webhookUrl` | Get each page pushed to your server. See [Webhooks](https://docs.spidra.io/api-reference/webhooks.md). |

Every field is in the [API reference](https://docs.spidra.io/api-reference/crawling/crawl.md).

## After the crawl

* **Extract again without crawling again.** [POST /crawl/\{jobId}/extract](https://docs.spidra.io/api-reference/crawling/crawl-extract.md) runs a new `transformInstruction` on the saved pages. It's quick and costs only the AI tokens. Good for trying different prompts.
* **Retry one page's extraction** with [POST /crawl/\{jobId}/retry/\{pageId}](https://docs.spidra.io/api-reference/crawling/crawl-retry-page.md) if it came back empty.
* **Download everything** as a ZIP with each page's data, Markdown and HTML.
* **Cancel** a running crawl with `DELETE /crawl/{jobId}`. Pages already done are kept.

## Cost

1 credit per page returned, plus AI credits when you extract, plus 5 per CAPTCHA solved. Navigation pages, failed pages and failed crawls are free. Re-extracting and page retries cost only AI tokens.

When your balance can't cover `maxPages`, the crawl isn't refused: it runs with `maxPages` lowered to what your credits cover, and the response says so (`maxPagesRequested` and a `message`). Credits held for pages that weren't needed come back at the end.

## In the dashboard

Open **Crawl** in the sidebar: **Website**, **What pages do you want?**, **What to extract** (with **Use a schema**), and **Options** for **Max pages**, **Max depth**, Stealth Mode, Fast mode and **More settings** (path filters, sitemap, subdomains, robots.txt, webhook and login cookies). **See its pages first** opens [Map](https://docs.spidra.io/map.md) for the same site. Results show as pages finish, and **How this crawl worked** shows the report above them.

![The Crawl page with the form on the left and three book pages in the results on the right](https://mintcdn.com/spidra/Ew-SZt3zJq_nwjCD/images/crawl-playground.webp?fit=max\&auto=format\&n=Ew-SZt3zJq_nwjCD\&q=85\&s=b941e830a83328fe8a5bc7e11d457c0c)

## Tips

* Start with `maxPages` of 3 to 5 to check your instruction, then raise it.
* Start the crawl at the section you want (`/blog`, `/docs`, a category page).
* Use the instruction to steer and path filters to fence.
* Read the report before raising `maxPages` when pages are missing.

## Next

> **[Map](https://docs.spidra.io/map.md)**
>
> See a site's pages first.

> **[Build a RAG knowledge base](https://docs.spidra.io/guides/rag-from-docs-site.md)**
>
> Crawl a docs site into clean Markdown.

> **[Structured output](https://docs.spidra.io/options/structured-output.md)**
>
> Schemas that keep every page the same.

> **[API reference](https://docs.spidra.io/api-reference/crawling/crawl.md)**
>
> Every field and response.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.