Batch scrape

Batch scrape takes up to 50 URLs and gives each one its own result. Use it when you have a list of pages with the same kind of data: products, articles, listings, profiles. Each URL succeeds or fails on its own, and failed ones can be retried in one call. Don’t have the URLs yet? Map a site to find them, or Crawl it.

Quick example

Get the title and price from two book pages, plus one URL that doesn’t exist:

curl -X POST https://api.spidra.io/api/batch/scrape \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "https://this-domain-does-not-exist-spidra-test.invalid"
    ],
    "prompt": "Extract the book details",
    "schema": {
      "type": "object",
      "properties": { "title": { "type": "string" }, "price": { "type": "number" } }
    },
    "wait": true
  }'

const batch = await spidra.batchScrape({
  urls: [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
  ],
  prompt: "Extract the book details",
  schema: { type: "object", properties: { title: { type: "string" }, price: { type: "number" } } },
});

for (const item of batch.items) console.log(item.status, item.result);
// completed { title: 'A Light in the Attic', price: 51.77 }
// completed { title: 'Tipping the Velvet', price: 53.74 }

batch = client.batch_scrape(
    [
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    ],
    prompt="Extract the book details",
    schema={"type": "object", "properties": {"title": {"type": "string"}, "price": {"type": "number"}}},
)

for item in batch.items:
    print(item.status, item.result)
# completed {'title': 'A Light in the Attic', 'price': 51.77}
# completed {'title': 'Tipping the Velvet', 'price': 53.74}

With "wait": true and up to 10 URLs, the result comes back in the same request if it finishes within 45 seconds. This one took 8 seconds (items shortened):

{
  "batchId": "38ae74ab-3a7f-421f-9d02-50c6b48d8c49",
  "status": "completed",
  "totalUrls": 3,
  "completedCount": 2,
  "failedCount": 1,
  "items": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "status": "completed",
      "result": { "title": "A Light in the Attic", "price": 51.77 },
      "error": null,
      "creditsUsed": 2
    },
    {
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "status": "completed",
      "result": { "title": "Tipping the Velvet", "price": 53.74 },
      "error": null,
      "creditsUsed": 2
    },
    {
      "url": "https://this-domain-does-not-exist-spidra-test.invalid",
      "status": "failed",
      "result": null,
      "error": "Failed to scrape https://this-domain-does-not-exist-spidra-test.invalid: the page could not be reached (ERR_TUNNEL_CONNECTION_FAILED)",
      "creditsUsed": 0
    }
  ],
  "createdAt": "2026-10-25T03:09:37.000Z",
  "finishedAt": "2026-10-25T03:09:45.000Z"
}

Without wait, or with more than 10 URLs, you get 202 with a batchId and poll GET /batch/scrape/{batchId}. The SDKs poll for you.

What you get back

status is completed once every URL is done, even if some failed. Check failedCount and each item’s status.
items has one entry per URL with its own status, result, error and creditsUsed. With a prompt or schema, result is the parsed data; without one, it’s the page as Markdown.
Items from supported sites also have structuredData with the site’s own fields.
completedCount and failedCount count up while it runs, so you can show progress.

Options

The same options as Scrape, applied to every URL: prompt, output, schema, extractContentOnly, screenshot, fullPageScreenshot, scrapeMode, useProxy, proxyCountry and cookies. Plus wait. urls takes plain strings or { "url": "..." } objects. Per-URL browser actions aren’t available in a batch; use Scrape for pages that need them.

Retry the URLs that failed

curl -X POST https://api.spidra.io/api/batch/scrape/38ae74ab-3a7f-421f-9d02-50c6b48d8c49/retry \
  -H "Authorization: Bearer $SPIDRA_API_KEY"

{ "status": "queued", "retriedCount": 1 }

Only failed items run again, with the batch’s original options. Cookies are never stored, so send cookies in the retry body again if the batch used them.

Cancel

DELETE /batch/scrape/{batchId} stops the items that haven’t started and gives back the credits held for them. The response says how many items it stopped (cancelledItems) and how many credits came back (creditsRefunded). Items already done keep their results.

Cost

Each URL costs what a scrape of that URL costs: 1 credit, plus AI credits with a prompt or schema (2 per book page above). Failed items cost nothing. See How credits work.

Limits

50 URLs per batch.
20 batches a minute, and up to 50 of your batch URLs unfinished at once (counted together with scrapes).
wait only for 10 URLs or fewer.

In the dashboard

Batches are started from the API, an SDK, the MCP server or a template. Every batch shows in Logs under Batch, where you can open it to see each URL’s result, status and cost.

Map

Find the URLs to batch.

Scrape a list of URLs to CSV

A complete script.

Structured output

Schemas for exact fields.

API reference

Every field and response.

Get started

Tools

Ready-made data

Getting the data you want

Supported sites

Guides

Dashboard

Account and billing

Quick example

What you get back

Options

Retry the URLs that failed

Cancel

Cost

Limits

In the dashboard

Next

Map

Scrape a list of URLs to CSV

Structured output

API reference

​Quick example

​What you get back

​Options

​Retry the URLs that failed

​Cancel

​Cost

​Limits

​In the dashboard

​Next

Map

Scrape a list of URLs to CSV

Structured output

API reference

Quick example

What you get back

Options

Retry the URLs that failed

Cancel

Cost

Limits

In the dashboard

Next