> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Rate Limits

> Understand API rate limits and how to handle them

Spidra enforces rate limits to ensure fair usage and platform stability. This page explains the limits and best practices for handling them.

<Note>The API allows up to **300 requests per IP address** every 15 minutes.</Note>

## Two Types of Limits

Spidra enforces two separate limits. Understanding the difference matters when building integrations.

### 1. Request Rate Limit (HTTP 429)

Controls how fast you can hit the API — up to 300 HTTP requests per IP per 15 minutes. This applies to all endpoints including polling.

Response:

```json theme={null}
{
  "status": "error",
  "message": "Too many requests, please try again later."
}
```

Includes standard headers:

| Header                | Description                            |
| --------------------- | -------------------------------------- |
| `RateLimit-Limit`     | Maximum requests allowed in the window |
| `RateLimit-Remaining` | Requests remaining in current window   |
| `RateLimit-Reset`     | Unix timestamp when the limit resets   |

### 2. Concurrent Job Limit (HTTP 429 with code `TOO_MANY_PENDING_JOBS`)

Controls how many jobs you can have queued at the same time. This is separate from how fast you call the API — it's about how many jobs are sitting in the queue waiting to run.

| Endpoint                | Limit           |
| ----------------------- | --------------- |
| `POST /scrape` (single) | 10 pending jobs |
| `POST /batch-scrape`    | 50 pending jobs |
| `POST /crawl`           | 5 pending jobs  |

Response:

```json theme={null}
{
  "status": "error",
  "message": "You have too many pending scrape jobs. Please wait for some to complete before submitting more.",
  "code": "TOO_MANY_PENDING_JOBS"
}
```

Wait for your current jobs to finish, then resubmit. You can check job status via `GET /scrape/{jobId}` or `GET /crawl/{jobId}`.

### 3. Service Capacity (HTTP 503 with code `SERVICE_BUSY`)

When the platform is under heavy load from all users combined, new jobs are temporarily rejected to protect stability. This is rare but can occur during traffic spikes.

Response:

```json theme={null}
{
  "status": "error",
  "message": "The scraping service is currently at capacity. Please retry in a few minutes.",
  "code": "SERVICE_BUSY",
  "retry_after": 60
}
```

Retry after the number of seconds in `retry_after`.

## Handling All Limits in Code

```javascript theme={null}
async function submitJob(request, maxRetries = 10) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch('https://api.spidra.io/api/scrape', {
      method: 'POST',
      headers: {
        'x-api-key': 'YOUR_API_KEY',
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(request)
    });

    if (response.ok) return response.json();

    const body = await response.json();

    if (response.status === 503 || body.code === 'SERVICE_BUSY') {
      // Platform at capacity — wait and retry
      const wait = (body.retry_after ?? 60) * 1000;
      await new Promise(r => setTimeout(r, wait));
      continue;
    }

    if (response.status === 429 && body.code === 'TOO_MANY_PENDING_JOBS') {
      // Your jobs are backed up — wait for them to drain
      await new Promise(r => setTimeout(r, 30_000));
      continue;
    }

    if (response.status === 429) {
      // HTTP rate limit — exponential backoff
      const wait = Math.pow(2, attempt) * 1000;
      await new Promise(r => setTimeout(r, wait));
      continue;
    }

    throw new Error(`Request failed: ${body.message}`);
  }
  throw new Error('Max retries reached');
}
```

### 2. Batch URLs Efficiently

Instead of making multiple single-URL requests, batch up to your plan's limit:

```json theme={null}
{
  "urls": [
    {"url": "https://example.com/page1"},
    {"url": "https://example.com/page2"},
    {"url": "https://example.com/page3"}
  ],
  "prompt": "Extract the title and main content"
}
```

### 3. Use Crawl for Large Sites

For scraping many pages from the same domain, use the [Crawl API](/api-reference/crawling/crawl) instead of multiple scrape requests. One crawl request can process up to 50 pages.

### 4. Cache Results

Store scrape results locally to avoid redundant requests for the same content.

<Note>Some limits vary depending on your plan. The number of concurrent URLs per scrape request and the number of actions per URL both increase on higher tiers. See the [Plans and Pricing](/billing/plans) page for a full breakdown.</Note>

## Increasing Limits

Need higher limits? [Contact us](mailto:hello@spidra.io) to discuss Enterprise plans with custom rate limits and dedicated infrastructure.