Skip to main content

​
The tools

Monitors run a template, a preset or a list of pages on a schedule and tell you when something changed.

​
Jobs and polling

Most work takes from a few seconds to a few minutes, so it runs as a job:
  1. You send the request, for example POST /scrape. The answer is 202 with a jobId (a batchId for batch scrapes, a runId for templates).
  2. You ask for the job, for example GET /scrape/{jobId}, every few seconds. Asking is free and isn’t rate limited.
  3. You stop when status is completed, failed or cancelled. Until then the status says the job is waiting or working (queued, pending, active or running, depending on the tool), and scrapes, crawls and parse jobs add a progress message.
Some requests can wait for the result instead:
  • Map always answers in the same request.
  • Search with "wait": true answers in the same request if it finishes within 20 seconds.
  • Batch scrape with "wait": true and up to 10 URLs answers in the same request if it finishes within 45 seconds.
  • Parse waits up to 20 seconds by default.
  • Template runs with "wait": true wait like the tool behind them.
If it doesn’t finish in time you get the usual 202 and a job ID, so code that waits must also handle polling. The Node.js and Python SDKs do all of this for you: scrape(), crawl(), search() and the rest return the finished result.

​
What comes back

A finished scrape always has the same outer shape:
  • result.content is the answer.
  • result.data has one entry per URL with that page’s Markdown.
  • result.stats says how long it took and how much AI it used.
What’s in content depends on what you ask for: Use a schema whenever code reads the result. Without one, field names and types can change between pages and runs. Batch scrapes, crawls, searches, parse jobs and template runs have their own result shapes, shown on each tool’s page.

​
How pages are read

By default Spidra opens each page in a real browser, so pages built with JavaScript work. You can change how a page is read:
  • Browser actions click, type, scroll and loop over items before the page is read.
  • "scrapeMode": "fast" reads the page over HTTP without a browser. It’s quicker for static pages; Spidra still uses a browser when a page needs one.
  • Stealth Mode ("useProxy": true) sends the request through residential proxies in the country you choose. CAPTCHAs are solved automatically either way.
  • Cookies let you read pages behind your own login.
  • Links from supported sites are read directly and come back with the site’s own fields.

​
Credits

You pay in credits. The main rules:
  • 1 credit per page scraped, crawled or parsed.
  • About 1 more credit per page when AI extracts data (it depends on the page’s length).
  • 1 credit per map, and 1 credit per 10 search results.
  • 5 credits per CAPTCHA solved.
  • Stealth Mode uses proxy bandwidth instead of credits.
Failed work is free. A page that can’t be read, comes back blank, needs a login or answers with an HTTP error isn’t charged. Neither is a map that finds nothing, a search with no results, or a crawl’s navigation pages (pages read only to find the ones you asked for). When you submit a job, Spidra holds the most it could cost and gives back the difference when it finishes. If you don’t have enough credits for the hold, the request is refused with 403 before anything runs. How credits work has the full rules and examples.

​
History

Every scrape, batch, crawl, map, search and parse is saved with its inputs, result and cost. Open them in the dashboard under Logs, or through the API with the log endpoints.

​
Limits

Every plan has every feature. Plans differ in credits, proxy bandwidth, pages per crawl and support. Request rates are on the Rate limits page.