Skip to main content
POST
Crawl

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json
baseUrl
string<uri>
required

The page to start from, starting with http:// or https://. Start from a page that links to the pages you want, such as a category or index page. Private and internal addresses are rejected.

Example:

"https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"

crawlInstruction
string
required

Which pages to return, in plain English, such as "Individual book pages". The report shows how it was understood.

transformInstruction
string

What to extract from each returned page, in plain English, such as "Return the book's title and whether it is in stock". Without a schema, the AI picks the shape.

schema
object

A JSON Schema for each page's data. Takes precedence over transformInstruction for the shape.

maxPages
integer
default:5

The most pages to return. Your plan sets the top value (100 on Free, 1,000 on Starter, 5,000 on Builder, 10,000 on Pro and Enterprise).

Required range: x >= 1
maxDepth
integer

How many clicks from the start page to follow. 0 returns only the start page. No limit by default.

Required range: x >= 0
includePaths
string[]

Only follow and return URLs whose path matches one of these. Globs such as "/blog/*" or regexes such as "^/docs/v[0-9]+/". A pattern starting with / matches from the start of the path.

excludePaths
string[]

Never follow or return URLs whose path matches one of these. Same pattern rules as includePaths.

allowSubdomains
boolean
default:false

Also follows links to subdomains of the start URL's host, such as blog.example.com from example.com.

crawlEntireDomain
boolean
default:false

Follows links anywhere on the same site, not only under the start URL's path.

ignoreQueryParams
boolean
default:false

Treats URLs that differ only by their query string as one page.

useSitemap
boolean
default:true

Also reads the site's sitemap for pages to consider.

respectRobots
boolean
default:false

Skips URLs the site's robots.txt asks crawlers not to visit.

scrapeMode
enum<string>
default:default

default opens every page in a real browser. fast fetches pages directly and opens a browser only when a page needs one. crawlerMode is accepted as an older name for this field.

Available options:
default,
fast
useProxy
boolean
default:false

Turns on Stealth Mode. Billed by bandwidth, not credits. Not needed when you set proxyCountry. false keeps the proxy off, even with a country. When Stealth Mode can't be used because your proxy bandwidth ran out, the request runs without it and warnings says so.

proxyCountry
string

The country to read pages from, as a two-letter code such as us, gb or de, or eu or asia. Setting a country turns Stealth Mode on (billed by bandwidth), and the response's warnings says so, unless useProxy is false: then the country isn't used, and warnings says that instead. global means any country.

Example:

"gb"

cookies
string

Cookies from your own signed-in browser session, to crawl pages that need an account. Never stored. See Logged-in pages.

webhookUrl
string<uri>

An http or https URL to notify for each page and when the crawl ends. Private and internal addresses are rejected. See Webhooks.

Response

The crawl was queued. Poll GET /crawl/{jobId} for the result.

status
enum<string>
Available options:
queued
jobId
string<uuid>

The crawl's ID. Use it with GET /crawl/{jobId}.

message
string

Where to poll, and whether maxPages was lowered.

maxPages
integer

Only when your credits covered fewer pages than you asked for. The maxPages the crawl runs with.

maxPagesRequested
integer

Only with maxPages. What you asked for.

deduplicated
boolean

true when the same request was sent in the last 5 seconds and this is that job's ID.