Scrape

Scrape reads 1 to 3 pages in a real browser and returns them as Markdown, or extracts the data you describe. With several URLs, the answer combines what’s on all of them. For a separate result per URL, use Batch scrape; for a whole site, use Crawl.

Quick example

Get the title and price of every book on a page:

curl -X POST https://api.spidra.io/api/scrape \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [{ "url": "https://books.toscrape.com" }],
    "prompt": "Return the title and price of every book on the page",
    "output": "json"
  }'

import { SpidraClient } from "spidra";

const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY });

const result = await spidra.scrape({
  urls: [{ url: "https://books.toscrape.com" }],
  prompt: "Return the title and price of every book on the page",
  output: "json",
});

import os
from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

result = client.scrape(
    "https://books.toscrape.com",
    prompt="Return the title and price of every book on the page",
    output="json",
)

The API answers with a jobId. Poll GET /scrape/{jobId} until status is completed; the SDKs do this for you and return the result:

{
  "status": "completed",
  "result": {
    "content": [
      { "title": "A Light in the Attic", "price": "£51.77" },
      { "title": "Tipping the Velvet", "price": "£53.74" }
    ],
    "data": [
      { "url": "https://books.toscrape.com", "title": "All products | Books to Scrape - Sandbox", "markdownContent": "...", "success": true }
    ],
    "stats": { "durationMs": 37458, "inputTokens": 2628, "outputTokens": 418, "totalTokens": 3046 }
  }
}

The list had all 20 books (shortened here). It cost 3 credits: 1 for the page and 2 for the AI.

What you get back

result.content is the answer. Without a prompt it’s the page as Markdown; with a prompt or schema it’s what you asked for.
result.data has one entry per URL, in the order you sent them, with the page’s markdownContent, title and success. Links from supported sites also have structuredData with the site’s own fields.
result.stats has the run time and the AI tokens used.
result.screenshots has image links when you ask for screenshots.

Raw Markdown, a prompt, or a schema

You send	`content`	Cost
Only the URL	The page as clean Markdown	1 credit
`prompt`	What you described, in a shape the AI picks	1 credit plus AI
`prompt` and `schema`	Exactly your fields and types, `null` for anything missing	1 credit plus AI

Without a schema, field names and types can change between runs: prices come back as "£51.77" here. With one, you get the same shape every time:

{
  "urls": [{ "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" }],
  "prompt": "Extract the book details",
  "schema": {
    "type": "object",
    "required": ["title", "price"],
    "properties": {
      "title": { "type": "string" },
      "price": { "type": "number" },
      "currency": { "type": ["string", "null"] },
      "in_stock": { "type": ["boolean", "null"] },
      "rating": { "type": ["string", "null"] }
    }
  }
}

{ "title": "A Light in the Attic", "price": 51.77, "currency": "£", "in_stock": true, "rating": null }

rating is null because that page shows its stars as an image, not text. See Structured output for writing schemas.

Options

Option	What it does
`prompt`	What to extract, in plain words.
`output`	`"markdown"` (default) or `"json"`.
`schema`	A JSON Schema for the result. Forces JSON output.
`actions` (per URL)	Click, type, scroll, wait or loop over items before the page is read. See Browser actions.
`extractContentOnly`	Drop menus, headers, footers and sidebars before reading the page.
`screenshot`, `fullPageScreenshot`	Capture the visible page, or the whole page.
`scrapeMode`	`"fast"` reads the page over HTTP without a browser. Quicker for static pages; actions and screenshots are off.
`useProxy`, `proxyCountry`	Stealth Mode: send the request through residential proxies, in a country such as `"us"` or `"de"`. See Proxies and CAPTCHAs.
`cookies`	Read pages behind your own login. See Logged-in pages.

Every field is in the API reference.

Several URLs

Send up to 3 URLs to answer one question from several pages, such as comparing two products:

{
  "urls": [
    { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" },
    { "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html" }
  ],
  "prompt": "Which of these two books is cheaper, and by how much?"
}

The AI reads both pages and gives one answer in content. Each page’s Markdown is still in data. Without a prompt, content joins the pages, each starting with a Content from: <url> line.

Documents

Point Scrape at a PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB or CSV link and you get the document as Markdown. Scanned pages are read with AI. For uploads, page ranges, a cost estimate first, or many documents, use Parse. See Documents by URL.

How it works

Each page opens in a real browser, so pages built with JavaScript work. Cookie banners are dismissed and CAPTCHAs are solved automatically.
A page that can’t be read fails the job with a plain error, such as “the page answered HTTP 404”, and costs nothing. So does a page that comes back blank or behind a sign-in wall.
Links from supported sites are read directly, without a browser, and come back with the site’s own fields.
Sending the same scrape twice within a few seconds returns the same job.

Cost

1 credit per URL, plus AI credits when you use a prompt or schema (usually 1 or 2 per page, more on long pages), plus 5 per CAPTCHA solved. Screenshots, actions and Fast Mode cost nothing extra. Stealth Mode uses proxy bandwidth. Failed jobs are free. See How credits work.

In the dashboard

Open Scrape in the sidebar. The form has the same options: up to 3 URLs (each with Add action for browser actions), Extraction instruction, Output (Markdown, JSON or Screenshot), Define schema, Stealth mode, Extract content only, Authentication for cookies, and the Fast mode switch. Code shows the request for your settings in curl, JavaScript or Python, and Save as preset keeps the setup for later.

The Scrape page with the form on the left and the output on the right

When the result isn’t what you expected

content is empty or {}. The data may load after the page opens. Add a scroll or wait action, or take a screenshot to see what Spidra saw.
A field is null. It isn’t in the page’s text. Check the page, then reword the prompt or the field.
The job failed. error says why. You weren’t charged.
The site blocks you. Turn on Stealth Mode, and try a country where the site is popular.

Batch scrape

Up to 50 URLs, one result each.

Structured output

Schemas for exact fields and types.

Browser actions

Click, scroll and loop before reading.

API reference

Every field and response.

Get started

Tools

Ready-made data

Getting the data you want

Supported sites

Guides

Dashboard

Account and billing

Quick example

What you get back

Raw Markdown, a prompt, or a schema

Options

Several URLs

Documents

How it works

Cost

In the dashboard

When the result isn’t what you expected

Next

Batch scrape

Structured output

Browser actions

API reference

​Quick example

​What you get back

​Raw Markdown, a prompt, or a schema

​Options

​Several URLs

​Documents

​How it works

​Cost

​In the dashboard

​When the result isn’t what you expected

​Next

Batch scrape

Structured output

Browser actions

API reference

Quick example

What you get back

Raw Markdown, a prompt, or a schema

Options

Several URLs

Documents

How it works

Cost

In the dashboard

When the result isn’t what you expected

Next