> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape a list of URLs to CSV

> Turn a list of product, article or listing URLs into a CSV with the same columns for every row, with failed pages reported and not charged.

You have a list of pages with the same kind of data on each, and you want one CSV row per page. This guide uses [Batch scrape](https://docs.spidra.io/batch-scrape.md) with a schema, so every row has the same columns, and writes the CSV in Python.

You need Python 3.9+, `pip install spidra`, and your API key in `SPIDRA_API_KEY`.

## 1. Describe the columns as a schema

The schema's properties become your CSV columns. Make anything that may be missing nullable:

```python theme={null}
schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "in_stock": {"type": ["boolean", "null"]},
        "upc": {"type": ["string", "null"]},
    },
}
```

## 2. Scrape the URLs and write the CSV

`batch_scrape()` sends up to 50 URLs and waits for all of them. Each item has its own `status`, so a page that fails doesn't stop the others:

```python theme={null}
# books_to_csv.py
import csv
import os

from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

urls = [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    "https://books.toscrape.com/catalogue/soumission_998/index.html",
    "https://books.toscrape.com/catalogue/sharp-objects_997/index.html",
    "https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html",
]

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "in_stock": {"type": ["boolean", "null"]},
        "upc": {"type": ["string", "null"]},
    },
}

batch = client.batch_scrape(urls, prompt="Extract the book details", schema=schema)

with open("books.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["url", "title", "price", "in_stock", "upc"])
    writer.writeheader()
    for item in batch.items:
        if item.status == "completed":
            writer.writerow({"url": item.url, **item.result})
        else:
            print(f"Failed: {item.url}\n  {item.error}")

print(f"{batch.completed_count} rows written, {batch.failed_count} failed")
```

Running it prints the page that doesn't exist and the totals:

```text theme={null}
Failed: https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html
  Failed to scrape https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html: The page was not delivered: the page answered HTTP 404. Nothing was charged. Check the URL.
4 rows written, 1 failed
```

And `books.csv`:

```text theme={null}
url,title,price,in_stock,upc
https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html,A Light in the Attic,51.77,True,a897fe39b1053632
https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html,Tipping the Velvet,53.74,True,90fa61229261140a
https://books.toscrape.com/catalogue/soumission_998/index.html,Soumission,50.1,True,6957f44c3847a760
https://books.toscrape.com/catalogue/sharp-objects_997/index.html,Sharp Objects,47.82,True,e00eb4fd7b871a48
```

Each book page cost 2 credits (1 for the page, 1 for the AI). The page that doesn't exist cost nothing.

## 3. More than 50 URLs

A batch takes up to 50 URLs. Send longer lists in chunks of 50 and append to the same file:

```python theme={null}
for start in range(0, len(urls), 50):
    batch = client.batch_scrape(urls[start:start + 50], prompt="Extract the book details", schema=schema)
    # write batch.items as above
```

You can send up to 20 batches a minute.

## If something fails

* **A page failed.** Its `error` says why, and it wasn't charged. [Handle failed pages and retries](https://docs.spidra.io/guides/handle-failures.md) shows how to retry just the failed items.
* **A column is empty.** The value isn't in the page's text. Check the page, then reword the prompt or the field name.
* **The site blocks you.** Add `use_proxy=True` and `proxy_country="us"` (or the site's country) to `batch_scrape()`.

## Variations

* No URLs yet? [Map](https://docs.spidra.io/map.md) the site first, or follow [Map a site and scrape only the pages you need](https://docs.spidra.io/guides/map-then-scrape.md).
* Links from [supported sites](https://docs.spidra.io/sites/overview.md) such as Amazon or Zillow don't need a schema: send `output="json"` with no prompt and you get the site's own fields, with no AI cost.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.