Scrape a list of URLs to CSV

You have a list of pages with the same kind of data on each, and you want one CSV row per page. This guide uses Batch scrape with a schema, so every row has the same columns, and writes the CSV in Python. You need Python 3.9+, pip install spidra, and your API key in SPIDRA_API_KEY.

1. Describe the columns as a schema

The schema’s properties become your CSV columns. Make anything that may be missing nullable:

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "in_stock": {"type": ["boolean", "null"]},
        "upc": {"type": ["string", "null"]},
    },
}

2. Scrape the URLs and write the CSV

batch_scrape() sends up to 50 URLs and waits for all of them. Each item has its own status, so a page that fails doesn’t stop the others:

# books_to_csv.py
import csv
import os

from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

urls = [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    "https://books.toscrape.com/catalogue/soumission_998/index.html",
    "https://books.toscrape.com/catalogue/sharp-objects_997/index.html",
    "https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html",
]

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "in_stock": {"type": ["boolean", "null"]},
        "upc": {"type": ["string", "null"]},
    },
}

batch = client.batch_scrape(urls, prompt="Extract the book details", schema=schema)

with open("books.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["url", "title", "price", "in_stock", "upc"])
    writer.writeheader()
    for item in batch.items:
        if item.status == "completed":
            writer.writerow({"url": item.url, **item.result})
        else:
            print(f"Failed: {item.url}\n  {item.error}")

print(f"{batch.completed_count} rows written, {batch.failed_count} failed")

Running it prints the page that doesn’t exist and the totals:

Failed: https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html
  Failed to scrape https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html: The page was not delivered: the page answered HTTP 404. Nothing was charged. Check the URL.
4 rows written, 1 failed

And books.csv:

url,title,price,in_stock,upc
https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html,A Light in the Attic,51.77,True,a897fe39b1053632
https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html,Tipping the Velvet,53.74,True,90fa61229261140a
https://books.toscrape.com/catalogue/soumission_998/index.html,Soumission,50.1,True,6957f44c3847a760
https://books.toscrape.com/catalogue/sharp-objects_997/index.html,Sharp Objects,47.82,True,e00eb4fd7b871a48

Each book page cost 2 credits (1 for the page, 1 for the AI). The page that doesn’t exist cost nothing.

3. More than 50 URLs

A batch takes up to 50 URLs. Send longer lists in chunks of 50 and append to the same file:

for start in range(0, len(urls), 50):
    batch = client.batch_scrape(urls[start:start + 50], prompt="Extract the book details", schema=schema)
    # write batch.items as above

You can send up to 20 batches a minute.

If something fails

A page failed. Its error says why, and it wasn’t charged. Handle failed pages and retries shows how to retry just the failed items.
A column is empty. The value isn’t in the page’s text. Check the page, then reword the prompt or the field name.
The site blocks you. Add use_proxy=True and proxy_country="us" (or the site’s country) to batch_scrape().

Variations

No URLs yet? Map the site first, or follow Map a site and scrape only the pages you need.
Links from supported sites such as Amazon or Zillow don’t need a schema: send output="json" with no prompt and you get the site’s own fields, with no AI cost.

Get started

Tools

Ready-made data

Getting the data you want

Supported sites

Guides

Dashboard

Account and billing

Scrape a list of URLs to CSV

1. Describe the columns as a schema

2. Scrape the URLs and write the CSV

3. More than 50 URLs

If something fails

Variations

​1. Describe the columns as a schema

​2. Scrape the URLs and write the CSV

​3. More than 50 URLs

​If something fails

​Variations

1. Describe the columns as a schema

2. Scrape the URLs and write the CSV

3. More than 50 URLs

If something fails

Variations