Skip to main content
Spidra enforces rate limits to ensure fair usage and platform stability. This page explains the limits and best practices for handling them.
The API allows up to 300 requests per IP address every 15 minutes.

​
Rate Limit Response

When you exceed the rate limit, you’ll receive a 429 Too Many Requests response: 
{
  "status": "error",
  "message": "Too many requests, please try again later."
}
The response includes standard rate limit headers:
HeaderDescription
RateLimit-LimitMaximum requests allowed in the window
RateLimit-RemainingRequests remaining in current window
RateLimit-ResetUnix timestamp when the limit resets

​
Best Practices

​
1. Implement Exponential Backoff

When you receive a 429 response, wait before retrying: 
async function scrapeWithRetry(request, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch('https://api.spidra.io/api/scrape', {
      method: 'POST',
      headers: {
        'x-api-key': 'YOUR_API_KEY',
        'Content-Type': 'application/json'
      },
      body: JSON.stringify(request)
    });

    if (response.status === 429) {
      const waitTime = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
      await new Promise(r => setTimeout(r, waitTime));
      continue;
    }

    return response.json();
  }
  throw new Error('Rate limit exceeded after retries');
}

​
2. Batch URLs Efficiently

Instead of making multiple single-URL requests, batch up to your plan’s limit: 
{
  "urls": [
    {"url": "https://example.com/page1"},
    {"url": "https://example.com/page2"},
    {"url": "https://example.com/page3"}
  ],
  "prompt": "Extract the title and main content"
}

​
3. Use Crawl for Large Sites

For scraping many pages from the same domain, use the Crawl API instead of multiple scrape requests. One crawl request can process up to 50 pages.

​
4. Cache Results

Store scrape results locally to avoid redundant requests for the same content.
Some limits vary depending on your plan. The number of concurrent URLs per scrape request and the number of actions per URL both increase on higher tiers. See the Plans and Pricing page for a full breakdown.

​
Increasing Limits

Need higher limits? Contact us to discuss Enterprise plans with custom rate limits and dedicated infrastructure.