> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Django

> Use Spidra's web scraping, crawling, search, and document parsing in Django with management commands, Django Signals, built-in URL routing, and async view support.

The Spidra Django integration (`spidra-django`) brings the full Spidra API into Django as a proper installed app: configure credentials in `settings.py`, include ready-made URL patterns in `urls.py`, receive webhook events as native Django Signals, and run scraping tasks from the terminal with `python manage.py spidra_*` management commands.

## Installation

```bash theme={null}
pip install spidra-django
```

Requires Python 3.9+ and Django 4.2+.

> [!NOTE]
>
> Get your API key from [app.spidra.io](https://app.spidra.io) under **API Keys**.
> Store it in an environment variable and reference it from `settings.py`. Never hardcode it.

## Setup

### 1. Add to `INSTALLED_APPS`

```python theme={null}
# settings.py
INSTALLED_APPS = [
    # ...
    "spidra_django.apps.SpidraConfig",  # or simply "spidra_django"
]
```

### 2. Configure in `settings.py`

```python theme={null}
import os

SPIDRA_API_KEY = os.environ.get("SPIDRA_API_KEY", "")
SPIDRA_WEBHOOK_SECRET = os.environ.get("SPIDRA_WEBHOOK_SECRET", "")

# Optional
SPIDRA_API_URL = "https://api.spidra.io/api"
SPIDRA_MAX_RETRIES = 3
SPIDRA_BACKOFF_FACTOR = 1.0
```

## Using the client

`spidra-django` exports a global `spidra` proxy that is ready to use anywhere in your project without passing configuration around:

```python theme={null}
from spidra_django import spidra

result = spidra.scrape("https://example.com", output="markdown")
print(result.content)
```

For factory-style access — useful in tests or multi-tenant setups — use `get_spidra()` and `get_async_spidra()`:

```python theme={null}
from spidra_django import get_spidra, get_async_spidra

client = get_spidra()
result = client.scrape("https://example.com")
```

## Scraping

Extract markdown or structured JSON from any URL, with browser rendering, CAPTCHA bypass, and residential proxies:

```python theme={null}
from django.http import JsonResponse
from spidra_django import spidra, spidra_json_response


def scrape_view(request):
    url = request.GET.get("url")
    result = spidra.scrape(url, output="markdown", use_proxy=True)
    return spidra_json_response({"content": result.content, "stats": result.stats})


def extract_product_view(request):
    url = request.POST.get("url")
    result = spidra.scrape(
        url,
        prompt="Extract product details",
        schema={
            "type": "object",
            "properties": {
                "title":    {"type": "string"},
                "price":    {"type": "number"},
                "in_stock": {"type": "boolean"},
            },
            "required": ["title", "price", "in_stock"],
        },
        output="json",
    )
    return spidra_json_response(result.content)
```

`spidra_json_response()` recursively serializes Spidra dataclasses and returns a `JsonResponse`. It accepts the same arguments as Django's `JsonResponse`, including `status`.

## Searching

```python theme={null}
def search_view(request):
    query = request.GET.get("q", "")
    limit = int(request.GET.get("limit", 10))
    results = spidra.search(query, sources=["web", "news"], limit=limit)
    return spidra_json_response(results)
```

Available `sources`: `"web"`, `"news"`, `"images"`, `"videos"`, `"research"`, `"developer"`. Defaults to `["web"]`.

## Crawling

```python theme={null}
def crawl_view(request):
    start_url = request.POST.get("url")
    job = spidra.crawl(
        start_url,
        crawl_instruction="Follow blog post links only",
        transform_instruction="Extract title, author, publish date, and summary",
        max_pages=30,
    )
    pages = [{"url": p.url, "data": p.data} for p in job.result]
    return spidra_json_response(pages)
```

For large crawls, submit without waiting:

```python theme={null}
queued = spidra.start_crawl(start_url, crawl_instruction="...", max_pages=100)
# later...
status = spidra.get_crawl(queued.job_id)
```

## Mapping

```python theme={null}
def map_view(request):
    url = request.GET.get("url")
    result = spidra.map(url, limit=100)
    return spidra_json_response([{"url": l.url} for l in result.links])
```

## Batch scraping

```python theme={null}
def batch_view(request):
    urls = request.POST.getlist("urls")
    batch = spidra.batch_scrape(urls, prompt="Extract title and meta description", output="json")
    items = [{"url": i.url, "result": i.result} for i in batch.items if i.status == "completed"]
    return spidra_json_response(items)
```

## Parsing documents

```python theme={null}
def parse_view(request):
    doc = spidra.parse("report.pdf", prompt="Invoice number, total amount, and due date")
    return spidra_json_response({"markdown": doc.markdown})
```

## Async views

Django 4.1+ supports native `async def` views. Use `spidra.async_client` for non-blocking calls:

```python theme={null}
from spidra_django import spidra, spidra_json_response


async def async_scrape_view(request):
    result = await spidra.async_client.scrape("https://example.com")
    return spidra_json_response(result)
```

## Webhooks

Spidra sends signed webhook events to your endpoint. Use `@spidra_webhook_required` to verify HMAC-SHA256 signatures automatically:

```python theme={null}
from django.views.decorators.csrf import csrf_exempt
from spidra_django import spidra_webhook_required
from django.http import JsonResponse
import json


@csrf_exempt
@spidra_webhook_required
def webhook_view(request):
    event = json.loads(request.body)
    print(f"Event: {event['event']}")
    return JsonResponse({"status": "ok"})
```

The decorator reads `SPIDRA_WEBHOOK_SECRET` from settings, verifies `X-Spidra-Signature` against the raw body, and returns a `401` response if the signature is invalid.

### Django Signals

`spidra-django` dispatches a native Django Signal — `spidra_webhook_received` — whenever a verified webhook event is received through the built-in URL routes:

```python theme={null}
from spidra_django.signals import spidra_webhook_received


def handle_webhook(sender, event, data, request, **kwargs):
    if event == "crawl.completed":
        job_id = data.get("jobId")
        print(f"Crawl {job_id} completed")

spidra_webhook_received.connect(handle_webhook)
```

Or use the `@receiver` decorator:

```python theme={null}
from django.dispatch import receiver
from spidra_django.signals import spidra_webhook_received


@receiver(spidra_webhook_received)
def on_spidra_webhook(sender, event, data, request, **kwargs):
    print(f"Received: {event}")
```

### Event listeners

Alternatively, register listeners on the `spidra` proxy object. These fire when `spidra.dispatch_event()` is called manually from your own webhook endpoint:

```python theme={null}
from spidra_django import spidra


@spidra.on("crawl.completed")
def on_crawl_done(data):
    print(f"Crawl done: {data.get('jobId')}")

@spidra.on("crawl.page")
def on_page(data):
    print(f"Page crawled: {data['page']['url']}")
```

## Built-in URL routing

Include the pre-built URL patterns to expose a complete Spidra REST API without writing any views:

```python theme={null}
# urls.py
from django.urls import include, path

urlpatterns = [
    # ...
    path("api/spidra/", include("spidra_django.urls")),
]
```

This exposes:

| Method | Endpoint | Description |
| - | - | - |
| `POST` | `/api/spidra/scrape/` | Scrape a page (`url`, `prompt`, `schema`, `renderJs`, `useProxy`) |
| `GET`/`POST` | `/api/spidra/search/` | Web search (`q`, `limit`, `sources`) |
| `POST` | `/api/spidra/map/` | Map domain links |
| `POST` | `/api/spidra/crawl/` | Start or execute crawl |
| `GET` | `/api/spidra/crawl/<id>/` | Crawl status and results |
| `GET` | `/api/spidra/crawl/<id>/report/` | Crawl skip report |
| `DELETE` | `/api/spidra/crawl/<id>/` | Cancel crawl |
| `POST` | `/api/spidra/crawl/<id>/retry/<page_id>/` | Retry one page's extraction |
| `POST` | `/api/spidra/batch/` | Batch scrape |
| `GET` | `/api/spidra/batch/<id>/` | Batch status |
| `DELETE` | `/api/spidra/batch/<id>/` | Cancel batch |
| `POST` | `/api/spidra/batch/<id>/retry/` | Retry failed items |
| `POST` | `/api/spidra/parse/` | Parse document |
| `POST` | `/api/spidra/parse/estimate/` | Estimate parse credits |
| `GET` | `/api/spidra/parse/<id>/` | Parse result |
| `DELETE` | `/api/spidra/parse/files/<id>/` | Delete uploaded file |
| `GET` | `/api/spidra/templates/` | List templates |
| `GET` | `/api/spidra/templates/<id>/` | Template details |
| `POST` | `/api/spidra/templates/<id>/run/` | Run template |
| `GET`/`POST` | `/api/spidra/presets/` | List or create presets |
| `GET`/`POST` | `/api/spidra/monitors/` | List or create monitors |
| `POST` | `/api/spidra/monitors/<id>/check/` | Trigger monitor check |
| `GET` | `/api/spidra/account/` | Account and credit info |
| `GET` | `/api/spidra/usage/` | Usage history |
| `POST` | `/api/spidra/webhooks/` | HMAC-verified webhook receiver (fires `spidra_webhook_received`) |

## Management commands

`spidra-django` adds management commands for automation scripts, cron jobs, and debugging:

```bash theme={null}
# Account and credits
python manage.py spidra_account

# Usage breakdown
python manage.py spidra_usage --range 7d

# Scrape a URL
python manage.py spidra_scrape https://news.ycombinator.com --format markdown --prompt "Top 3 headlines"

# Web search
python manage.py spidra_search "Django ASGI tutorials" --limit 5

# Map a domain
python manage.py spidra_map https://example.com --limit 20

# Crawl a site
python manage.py spidra_crawl https://example.com --max-pages 10
```

## Error handling

All Spidra exceptions inherit from `SpidraError` and are re-exported from `spidra_django`:

```python theme={null}
from django.http import JsonResponse
from spidra_django import (
    spidra,
    SpidraError,
    SpidraAuthenticationError,
    SpidraPaymentRequiredError,
    SpidraRateLimitError,
)


def safe_scrape_view(request):
    try:
        result = spidra.scrape(request.GET.get("url"))
        return JsonResponse({"content": result.content})
    except SpidraAuthenticationError:
        return JsonResponse({"error": "Invalid API key"}, status=401)
    except SpidraPaymentRequiredError:
        return JsonResponse({"error": "Spidra credits exhausted"}, status=402)
    except SpidraRateLimitError as e:
        return JsonResponse(
            {"error": "Rate limited", "retry_after": getattr(e, "retry_after", 5)},
            status=429,
        )
    except SpidraError as e:
        return JsonResponse({"error": str(e)}, status=500)
```

See the [Python SDK](https://docs.spidra.io/sdks/python.md#error-handling) for the full exception hierarchy.

> [card] [Flask](https://docs.spidra.io/sdks/flask.md)
>
> Flask extension with Blueprint routing, `flask spidra` CLI, and HMAC webhook decorator.

> [card] [FastAPI](https://docs.spidra.io/sdks/fastapi.md)
>
> Native async-first integration with dependency injection, Pydantic models, and OpenAPI docs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.