> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Handle failed pages and retries

> Tell failed pages from failed requests, retry only what can succeed, and back off when Spidra asks you to slow down. Failed pages are never charged.

Two different things can go wrong, and they're handled differently:

| | Example | What to do |
| - | - | - |
| **A page fails** | It answered 404, it's behind a sign-in, the site blocked it | The job or batch item has `status: "failed"` and an `error`. You aren't charged. Retry only if the reason is temporary. |
| **A request fails** | `429` (too many requests), `503` (Spidra is busy), `500` | The HTTP call itself fails. Wait and send it again. |

This guide handles both in Python, with a batch where one URL doesn't exist.

## Back off when Spidra asks you to

`429` and `503` mean "try again shortly". Retry with an increasing wait. In the Python SDK they raise `SpidraRateLimitError` and `SpidraServerError`:

```python theme={null}
def submit(urls, attempts=5):
    """Start a batch, backing off if Spidra says to slow down or is busy."""
    for attempt in range(attempts):
        try:
            return client.start_batch_scrape(urls, output="markdown")
        except (SpidraRateLimitError, SpidraServerError) as err:
            wait = 2 ** attempt
            print(f"{err.status}: retrying in {wait}s")
            time.sleep(wait)
    raise RuntimeError("Spidra stayed busy; try again later")
```

Don't retry other `4xx` errors unchanged. A `422` (bad input) or `403` (out of credits) fails the same way every time; [Errors](https://docs.spidra.io/api-reference/errors.md) lists each one.

## Find the pages that failed

A batch's `status` is `completed` when every URL is done, **including the failed ones**, so check each item:

```python theme={null}
# retries.py
import os
import time

from spidra import Spidra, SpidraRateLimitError, SpidraServerError

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

urls = [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html",
    "https://quotes.toscrape.com/",
]


def submit(urls, attempts=5):
    """Start a batch, backing off if Spidra says to slow down or is busy."""
    for attempt in range(attempts):
        try:
            return client.start_batch_scrape(urls, output="markdown")
        except (SpidraRateLimitError, SpidraServerError) as err:
            wait = 2 ** attempt
            print(f"{err.status}: retrying in {wait}s")
            time.sleep(wait)
    raise RuntimeError("Spidra stayed busy; try again later")


def wait_for(batch_id):
    while True:
        time.sleep(3)
        batch = client.get_batch_scrape(batch_id)
        if batch.status in ("completed", "cancelled"):
            return batch


batch_id = submit(urls).batch_id
batch = wait_for(batch_id)
print(f"{batch.completed_count} done, {batch.failed_count} failed")

for item in batch.items:
    if item.status == "failed":
        print(f"  {item.url}\n  {item.error}")

# Retry once: this fixes pages that failed for a passing reason (a timeout, a busy site)
if batch.failed_count:
    print(f"Retrying {client.retry_batch_scrape(batch_id)} item(s)")
    batch = wait_for(batch_id)
    print(f"{batch.completed_count} done, {batch.failed_count} failed")
```

```text theme={null}
2 done, 1 failed
  https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html
  Failed to scrape https://books.toscrape.com/catalogue/this-page-does-not-exist_000/index.html: The page was not delivered: the page answered HTTP 404. Nothing was charged. Check the URL.
Retrying 1 item(s)
2 done, 1 failed
```

The retry didn't help, and that's the point: the page doesn't exist. Neither the first try nor the retry was charged.

## Which failures are worth retrying

Read `error`:

| The error says | Retry? |
| - | - |
| The page answered HTTP 404 or 410, the domain couldn't be found | No. Fix the URL. |
| It needs a sign-in, or it's a page the site only shows signed-in users | No. Use [cookies](https://docs.spidra.io/options/logged-in-pages.md), or skip it. |
| The page answered 403, 429 or a bot check stayed on screen | Yes, once or twice, ideally with [Stealth Mode](https://docs.spidra.io/options/proxies-and-captchas.md) and a country where the site is popular. |
| A timeout, or the page couldn't be reached | Yes. |

`retry_batch_scrape()` re-runs only the failed items, with the batch's original options. Cookies are never stored, so pass them again if the batch used them.

## Single scrapes and crawls

* **A scrape** that fails answers `status: "failed"` with an `error`, and the SDK's `scrape()` raises `SpidraJobFailedError`. Send it again if the reason is temporary.
* **A crawl** skips pages that fail and carries on; they're marked in the results and aren't charged. A crawl that can't read its start page fails as a whole, free.

## Good to know

* The SDKs already retry `502`, `503` and `504` responses and keep polling through brief network errors.
* Submitting a lot at once? Stay under the [rate limits](https://docs.spidra.io/api-reference/rate-limits.md): 60 scrapes and 20 batches a minute, and up to 10 unfinished scrapes at a time.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.