Django

The Spidra Django integration (spidra-django) brings the full Spidra API into Django as a proper installed app: configure credentials in settings.py, include ready-made URL patterns in urls.py, receive webhook events as native Django Signals, and run scraping tasks from the terminal with python manage.py spidra_* management commands.

Installation

pip install spidra-django

Requires Python 3.9+ and Django 4.2+.

Get your API key from app.spidra.io under API Keys. Store it in an environment variable and reference it from settings.py. Never hardcode it.

Setup

1. Add to `INSTALLED_APPS`

# settings.py
INSTALLED_APPS = [
    # ...
    "spidra_django.apps.SpidraConfig",  # or simply "spidra_django"
]

2. Configure in `settings.py`

import os

SPIDRA_API_KEY = os.environ.get("SPIDRA_API_KEY", "")
SPIDRA_WEBHOOK_SECRET = os.environ.get("SPIDRA_WEBHOOK_SECRET", "")

# Optional
SPIDRA_API_URL = "https://api.spidra.io/api"
SPIDRA_MAX_RETRIES = 3
SPIDRA_BACKOFF_FACTOR = 1.0

Using the client

spidra-django exports a global spidra proxy that is ready to use anywhere in your project without passing configuration around:

from spidra_django import spidra

result = spidra.scrape("https://example.com", output="markdown")
print(result.content)

For factory-style access — useful in tests or multi-tenant setups — use get_spidra() and get_async_spidra():

from spidra_django import get_spidra, get_async_spidra

client = get_spidra()
result = client.scrape("https://example.com")

Scraping

Extract markdown or structured JSON from any URL, with browser rendering, CAPTCHA bypass, and residential proxies:

from django.http import JsonResponse
from spidra_django import spidra, spidra_json_response


def scrape_view(request):
    url = request.GET.get("url")
    result = spidra.scrape(url, output="markdown", use_proxy=True)
    return spidra_json_response({"content": result.content, "stats": result.stats})


def extract_product_view(request):
    url = request.POST.get("url")
    result = spidra.scrape(
        url,
        prompt="Extract product details",
        schema={
            "type": "object",
            "properties": {
                "title":    {"type": "string"},
                "price":    {"type": "number"},
                "in_stock": {"type": "boolean"},
            },
            "required": ["title", "price", "in_stock"],
        },
        output="json",
    )
    return spidra_json_response(result.content)

spidra_json_response() recursively serializes Spidra dataclasses and returns a JsonResponse. It accepts the same arguments as Django’s JsonResponse, including status.

Searching

def search_view(request):
    query = request.GET.get("q", "")
    limit = int(request.GET.get("limit", 10))
    results = spidra.search(query, sources=["web", "news"], limit=limit)
    return spidra_json_response(results)

Available sources: "web", "news", "images", "videos", "research", "developer". Defaults to ["web"].

Crawling

def crawl_view(request):
    start_url = request.POST.get("url")
    job = spidra.crawl(
        start_url,
        crawl_instruction="Follow blog post links only",
        transform_instruction="Extract title, author, publish date, and summary",
        max_pages=30,
    )
    pages = [{"url": p.url, "data": p.data} for p in job.result]
    return spidra_json_response(pages)

For large crawls, submit without waiting:

queued = spidra.start_crawl(start_url, crawl_instruction="...", max_pages=100)
# later...
status = spidra.get_crawl(queued.job_id)

Mapping

def map_view(request):
    url = request.GET.get("url")
    result = spidra.map(url, limit=100)
    return spidra_json_response([{"url": l.url} for l in result.links])

Batch scraping

def batch_view(request):
    urls = request.POST.getlist("urls")
    batch = spidra.batch_scrape(urls, prompt="Extract title and meta description", output="json")
    items = [{"url": i.url, "result": i.result} for i in batch.items if i.status == "completed"]
    return spidra_json_response(items)

Parsing documents

def parse_view(request):
    doc = spidra.parse("report.pdf", prompt="Invoice number, total amount, and due date")
    return spidra_json_response({"markdown": doc.markdown})

Async views

Django 4.1+ supports native async def views. Use spidra.async_client for non-blocking calls:

from spidra_django import spidra, spidra_json_response


async def async_scrape_view(request):
    result = await spidra.async_client.scrape("https://example.com")
    return spidra_json_response(result)

Webhooks

Spidra sends signed webhook events to your endpoint. Use @spidra_webhook_required to verify HMAC-SHA256 signatures automatically:

from django.views.decorators.csrf import csrf_exempt
from spidra_django import spidra_webhook_required
from django.http import JsonResponse
import json


@csrf_exempt
@spidra_webhook_required
def webhook_view(request):
    event = json.loads(request.body)
    print(f"Event: {event['event']}")
    return JsonResponse({"status": "ok"})

The decorator reads SPIDRA_WEBHOOK_SECRET from settings, verifies X-Spidra-Signature against the raw body, and returns a 401 response if the signature is invalid.

Django Signals

spidra-django dispatches a native Django Signal — spidra_webhook_received — whenever a verified webhook event is received through the built-in URL routes:

from spidra_django.signals import spidra_webhook_received


def handle_webhook(sender, event, data, request, **kwargs):
    if event == "crawl.completed":
        job_id = data.get("jobId")
        print(f"Crawl {job_id} completed")

spidra_webhook_received.connect(handle_webhook)

Or use the @receiver decorator:

from django.dispatch import receiver
from spidra_django.signals import spidra_webhook_received


@receiver(spidra_webhook_received)
def on_spidra_webhook(sender, event, data, request, **kwargs):
    print(f"Received: {event}")

Event listeners

Alternatively, register listeners on the spidra proxy object. These fire when spidra.dispatch_event() is called manually from your own webhook endpoint:

from spidra_django import spidra


@spidra.on("crawl.completed")
def on_crawl_done(data):
    print(f"Crawl done: {data.get('jobId')}")

@spidra.on("crawl.page")
def on_page(data):
    print(f"Page crawled: {data['page']['url']}")

Built-in URL routing

Include the pre-built URL patterns to expose a complete Spidra REST API without writing any views:

# urls.py
from django.urls import include, path

urlpatterns = [
    # ...
    path("api/spidra/", include("spidra_django.urls")),
]

This exposes:

Method	Endpoint	Description
`POST`	`/api/spidra/scrape/`	Scrape a page (`url`, `prompt`, `schema`, `renderJs`, `useProxy`)
`GET`/`POST`	`/api/spidra/search/`	Web search (`q`, `limit`, `sources`)
`POST`	`/api/spidra/map/`	Map domain links
`POST`	`/api/spidra/crawl/`	Start or execute crawl
`GET`	`/api/spidra/crawl/<id>/`	Crawl status and results
`GET`	`/api/spidra/crawl/<id>/report/`	Crawl skip report
`DELETE`	`/api/spidra/crawl/<id>/`	Cancel crawl
`POST`	`/api/spidra/crawl/<id>/retry/<page_id>/`	Retry one page’s extraction
`POST`	`/api/spidra/batch/`	Batch scrape
`GET`	`/api/spidra/batch/<id>/`	Batch status
`DELETE`	`/api/spidra/batch/<id>/`	Cancel batch
`POST`	`/api/spidra/batch/<id>/retry/`	Retry failed items
`POST`	`/api/spidra/parse/`	Parse document
`POST`	`/api/spidra/parse/estimate/`	Estimate parse credits
`GET`	`/api/spidra/parse/<id>/`	Parse result
`DELETE`	`/api/spidra/parse/files/<id>/`	Delete uploaded file
`GET`	`/api/spidra/templates/`	List templates
`GET`	`/api/spidra/templates/<id>/`	Template details
`POST`	`/api/spidra/templates/<id>/run/`	Run template
`GET`/`POST`	`/api/spidra/presets/`	List or create presets
`GET`/`POST`	`/api/spidra/monitors/`	List or create monitors
`POST`	`/api/spidra/monitors/<id>/check/`	Trigger monitor check
`GET`	`/api/spidra/account/`	Account and credit info
`GET`	`/api/spidra/usage/`	Usage history
`POST`	`/api/spidra/webhooks/`	HMAC-verified webhook receiver (fires `spidra_webhook_received`)

Management commands

spidra-django adds management commands for automation scripts, cron jobs, and debugging:

# Account and credits
python manage.py spidra_account

# Usage breakdown
python manage.py spidra_usage --range 7d

# Scrape a URL
python manage.py spidra_scrape https://news.ycombinator.com --format markdown --prompt "Top 3 headlines"

# Web search
python manage.py spidra_search "Django ASGI tutorials" --limit 5

# Map a domain
python manage.py spidra_map https://example.com --limit 20

# Crawl a site
python manage.py spidra_crawl https://example.com --max-pages 10

Error handling

All Spidra exceptions inherit from SpidraError and are re-exported from spidra_django:

from django.http import JsonResponse
from spidra_django import (
    spidra,
    SpidraError,
    SpidraAuthenticationError,
    SpidraPaymentRequiredError,
    SpidraRateLimitError,
)


def safe_scrape_view(request):
    try:
        result = spidra.scrape(request.GET.get("url"))
        return JsonResponse({"content": result.content})
    except SpidraAuthenticationError:
        return JsonResponse({"error": "Invalid API key"}, status=401)
    except SpidraPaymentRequiredError:
        return JsonResponse({"error": "Spidra credits exhausted"}, status=402)
    except SpidraRateLimitError as e:
        return JsonResponse(
            {"error": "Rate limited", "retry_after": getattr(e, "retry_after", 5)},
            status=429,
        )
    except SpidraError as e:
        return JsonResponse({"error": str(e)}, status=500)

See the Python SDK for the full exception hierarchy.

Flask

Flask extension with Blueprint routing, flask spidra CLI, and HMAC webhook decorator.

FastAPI

Native async-first integration with dependency injection, Pydantic models, and OpenAPI docs.

SDKs

AI tools

Installation

Setup

1. Add to `INSTALLED_APPS`

2. Configure in `settings.py`

Using the client

Scraping

Searching

Crawling

Mapping

Batch scraping

Parsing documents

Async views

Webhooks

Django Signals

Event listeners

Built-in URL routing

Management commands

Error handling

Flask

FastAPI

​Installation

​Setup

​1. Add to INSTALLED_APPS

​2. Configure in settings.py

​Using the client

​Scraping

​Searching

​Crawling

​Mapping

​Batch scraping

​Parsing documents

​Async views

​Webhooks

​Django Signals

​Event listeners

​Built-in URL routing

​Management commands

​Error handling

Flask

FastAPI

Installation

Setup

1. Add to `INSTALLED_APPS`

2. Configure in `settings.py`

Using the client

Scraping

Searching

Crawling

Mapping

Batch scraping

Parsing documents

Async views

Webhooks

Django Signals

Event listeners

Built-in URL routing

Management commands

Error handling