Rate Limits

Spidra enforces rate limits to ensure fair usage and platform stability. This page explains the limits and best practices for handling them.

Submission limits are enforced per user account and are the same on every plan: 60 submissions per minute for scrape, crawl and search, and 20 per minute for batch scrape. Polling (GET) endpoints are not rate-limited.

Two Types of Limits

Spidra enforces two separate limits. Understanding the difference matters when building integrations.

1. Submission Rate Limit (HTTP 429)

Controls how fast you can submit new jobs. Each submit endpoint has its own counter per user account:

Endpoint	Limit
`POST /scrape`	60 per minute
`POST /crawl` (also crawl extract and page retry)	60 per minute
`POST /search`	60 per minute
`POST /batch/scrape`	20 per minute

Polling an existing job’s status is exempt from this limit. Response (429):

{
  "status": "error",
  "message": "Too many job submissions. Please slow down and try again shortly."
}

The batch endpoint answers with "Too many batch job submissions. Please slow down and try again shortly." instead. Submit endpoints return the limit state on every response, including errors, using the standard RateLimit and RateLimit-Policy headers:

RateLimit-Policy: "20-in-1min"; q=20; w=60
RateLimit: "20-in-1min"; r=19; t=60

Part	Meaning
`q`	The quota: how many submissions are allowed in the window
`w`	The window length in seconds
`r`	How many submissions you have left in the current window
`t`	Seconds until the window resets

2. Service Capacity (HTTP 503 with code `SERVICE_BUSY`)

When the platform is under heavy load from all users combined, new jobs are temporarily rejected to protect stability. This is rare but can occur during traffic spikes. Response:

{
  "status": "error",
  "message": "The scraping service is currently at capacity. Please retry in a few minutes.",
  "code": "SERVICE_BUSY",
  "retry_after": 60
}

Retry after the number of seconds in retry_after.

Handling All Limits in Code

async function submitJob(request, maxRetries = 10) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch('https://api.spidra.io/api/scrape', {
      method: 'POST',
      headers: {
        Authorization: 'Bearer YOUR_API_KEY',
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(request)
    });

    if (response.ok) return response.json();

    const body = await response.json();

    if (response.status === 503 || body.code === 'SERVICE_BUSY') {
      // Platform at capacity — wait and retry
      const wait = (body.retry_after ?? 60) * 1000;
      await new Promise(r => setTimeout(r, wait));
      continue;
    }

    if (response.status === 429) {
      // HTTP rate limit — exponential backoff
      const wait = Math.pow(2, attempt) * 1000;
      await new Promise(r => setTimeout(r, wait));
      continue;
    }

    throw new Error(`Request failed: ${body.message}`);
  }
  throw new Error('Max retries reached');
}

2. Batch URLs Efficiently

Instead of making multiple single-URL requests, send up to 3 URLs in one scrape request, or use Batch Scrape for up to 50:

{
  "urls": [
    {"url": "https://example.com/page1"},
    {"url": "https://example.com/page2"},
    {"url": "https://example.com/page3"}
  ],
  "prompt": "Extract the title and main content"
}

3. Use Crawl for Large Sites

For scraping many pages from the same domain, use the Crawl API instead of multiple scrape requests. One crawl request can return up to 10,000 pages, depending on your plan.

4. Cache Results

Store scrape results locally to avoid redundant requests for the same content.

Rate limits, the number of URLs per request (3 for scrape, 50 for batch, up to 50 pages for crawl) and the number of browser actions per URL are the same on every plan. Plans differ in monthly credits, proxy bandwidth and support. See Plans and Pricing.

Increasing Limits

Need higher limits? Contact us to talk about an Enterprise plan.

Using the API

Scrape Endpoints

Parse Endpoints

Search Endpoints

Crawl Endpoints

Account

Two Types of Limits

1. Submission Rate Limit (HTTP 429)

2. Service Capacity (HTTP 503 with code `SERVICE_BUSY`)

Handling All Limits in Code

2. Batch URLs Efficiently

3. Use Crawl for Large Sites

4. Cache Results

Increasing Limits

​Two Types of Limits

​1. Submission Rate Limit (HTTP 429)

​2. Service Capacity (HTTP 503 with code SERVICE_BUSY)

​Handling All Limits in Code

​2. Batch URLs Efficiently

​3. Use Crawl for Large Sites

​4. Cache Results

​Increasing Limits

Two Types of Limits

1. Submission Rate Limit (HTTP 429)

2. Service Capacity (HTTP 503 with code `SERVICE_BUSY`)

Handling All Limits in Code

2. Batch URLs Efficiently

3. Use Crawl for Large Sites

4. Cache Results

Increasing Limits