Skip to main content
Spidra enforces rate limits to ensure fair usage and platform stability. This page explains the limits and best practices for handling them.

​
API Rate Limits

Limit TypeValueWindow
Requests per IP1,00015 minutes
URLs per scrape job1–5Per request (varies by plan)
Max pages per crawl50Per crawl job

​
Rate Limit Response

When you exceed the rate limit, you’ll receive a 429 Too Many Requests response: 
{
  "status": "error",
  "message": "Too many requests, please try again later."
}
The response includes standard rate limit headers:
HeaderDescription
RateLimit-LimitMaximum requests allowed in the window
RateLimit-RemainingRequests remaining in current window
RateLimit-ResetUnix timestamp when the limit resets

​
Best Practices

​
1. Implement Exponential Backoff

When you receive a 429 response, wait before retrying: 
async function scrapeWithRetry(request, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch('https://api.spidra.io/api/scrape', {
      method: 'POST',
      headers: {
        'x-api-key': 'YOUR_API_KEY',
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(request)
    });

    if (response.status === 429) {
      const waitTime = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
      await new Promise(r => setTimeout(r, waitTime));
      continue;
    }

    return response.json();
  }
  throw new Error('Rate limit exceeded after retries');
}

​
2. Batch URLs Efficiently

Instead of making multiple single-URL requests, batch up to your plan’s limit: 
{
  "urls": [
    {"url": "https://example.com/page1"},
    {"url": "https://example.com/page2"},
    {"url": "https://example.com/page3"}
  ],
  "prompt": "Extract the title and main content"
}

​
3. Use Crawl for Large Sites

For scraping many pages from the same domain, use the Crawl API instead of multiple scrape requests. One crawl request can process up to 50 pages.

​
4. Cache Results

Store scrape results locally to avoid redundant requests for the same content.

​
Increasing Limits

Need higher limits? Contact us to discuss Enterprise plans with custom rate limits and dedicated infrastructure.