> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Introduction

> REST API reference for Spidra — scrape, batch scrape, crawl, and browser automation endpoints with full request and response examples.

The Spidra API gives you programmatic access to web data. All endpoints share a common base URL, authentication scheme, and response format described on this page.

## Features

<CardGroup cols={3}>
  <Card title="Scrape" icon="code" href="/api-reference/scraping/scrape">
    Extract content from any webpage with browser automation and AI processing.
  </Card>

  <Card title="Batch Scrape" icon="layer-group" href="/api-reference/scraping/batch-scrape">
    Queue up to 50 URLs in one request and collect per-item results in parallel.
  </Card>

  <Card title="Crawl" icon="spider" href="/api-reference/crawling/crawl">
    Crawl entire websites and extract structured data from every page.
  </Card>
</CardGroup>

## Base URL

All API requests use this base URL:

```
https://api.spidra.io/api
```

<Note>
  Python's built-in `urllib` (its default `User-Agent` is `Python-urllib/3.x`) and Perl's `libwww-perl` are rejected before they reach Spidra, with a `403` that mentions `browser_signature_banned` (Cloudflare error 1010). If you see that, use the [official SDKs](/sdks/python), use `requests` or `httpx`, or set your own `User-Agent` header such as `my-app/1.0`. `curl`, `fetch`, `requests`, `httpx`, Go, Java and the other common clients are not affected.
</Note>

## Authentication

Include your API key in the standard `Authorization: Bearer` header.

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.spidra.io/api/scrape \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"urls": [{"url": "https://example.com"}]}'
  ```

  ```python Python theme={null}
  import requests

  response = requests.post(
      "https://api.spidra.io/api/scrape",
      headers={"Authorization": "Bearer YOUR_API_KEY"},
      json={"urls": [{"url": "https://example.com"}]}
  )
  ```

  ```javascript Node.js theme={null}
  const response = await fetch("https://api.spidra.io/api/scrape", {
    method: "POST",
    headers: {
      Authorization: "Bearer YOUR_API_KEY",
      "Content-Type": "application/json"
    },
    body: JSON.stringify({ urls: [{ url: "https://example.com" }] })
  });
  ```
</CodeGroup>

Get your API key from [app.spidra.io](https://app.spidra.io) → API Keys.

<Warning>
  Keep your API key secret. Never expose it in frontend code.
</Warning>

## Response Codes

| Code | Meaning | What it indicates |
| - | - | - |
| `200` | OK | Request completed successfully |
| `202` | Accepted | Job queued. Poll the status endpoint for results |
| `400` | Bad Request | The request conflicts with the current state (e.g. cancelling a finished job) |
| `401` | Unauthorized | API key is missing, invalid, or expired |
| `402` | Payment Required | Subscription payment overdue |
| `403` | Forbidden | Credits exhausted or plan limit reached |
| `404` | Not Found | The job or resource does not exist |
| `422` | Unprocessable Entity | The request body failed validation — the response includes an `errors` array |
| `429` | Too Many Requests | Rate limit hit. Wait and retry with backoff |
| `500` | Server Error | Something went wrong on our end |

## Error Format

All errors return this structure:

```json theme={null}
{
  "status": "error",
  "message": "Detailed error explanation"
}
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.