> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Flask

> Use Spidra's web scraping, crawling, search, and document parsing directly inside Flask applications with the official spidra-flask integration.

The Spidra Flask integration (`spidra-flask`) brings the full Spidra API — scrape, batch scrape, search, crawl, map, parse, templates, presets, and monitors — natively into Flask. It follows Flask extension conventions: direct initialization or the Application Factory pattern, a pre-built Blueprint exposing REST endpoints, an event dispatcher for webhooks, and a `flask spidra` CLI group for terminal use.

## Installation

```bash theme={null}
pip install spidra-flask
```

Requires Python 3.9+ and Flask 2.3+.

> [!NOTE]
>
> Get your API key from [app.spidra.io](https://app.spidra.io) under **API Keys**.
> Store it in an environment variable or your Flask app config. Never hardcode it.

## Configuration

Configure credentials via environment variables or directly in `app.config`:

```bash theme={null}
# .env
SPIDRA_API_KEY="spd_live_your_api_key"
SPIDRA_WEBHOOK_SECRET="whsec_your_signing_secret"

# Optional
SPIDRA_API_URL="https://api.spidra.io/api"
SPIDRA_MAX_RETRIES=3
SPIDRA_BACKOFF_FACTOR=1.0
```

Or in your Flask config:

```python theme={null}
app.config["SPIDRA_API_KEY"] = "spd_live_your_api_key"
app.config["SPIDRA_WEBHOOK_SECRET"] = "whsec_your_signing_secret"
```

Flask app config takes precedence over environment variables.

## Initialization

`spidra-flask` supports both styles.

### Direct initialization

```python theme={null}
from flask import Flask
from spidra_flask import Spidra

app = Flask(__name__)
spidra = Spidra(app)
```

### Application Factory pattern

```python theme={null}
from flask import Flask
from spidra_flask import Spidra

spidra = Spidra()  # no app yet

def create_app():
    app = Flask(__name__)
    app.config.from_prefixed_env()
    spidra.init_app(app)
    return app
```

`SpidraFlask` is an alias for `Spidra` if you prefer a more explicit name.

## Scraping

`spidra.scrape()` extracts content from a URL — plain markdown or structured JSON — with browser rendering, CAPTCHA bypass, and optional residential proxies:

```python theme={null}
from flask import request
from spidra_flask import Spidra, spidra_jsonify

app = Flask(__name__)
spidra = Spidra(app)

@app.route("/scrape", methods=["POST"])
def scrape():
    url = request.json["url"]
    result = spidra.scrape(url, output="markdown", use_proxy=True)
    return spidra_jsonify({"content": result.content, "stats": result.stats})
```

For structured extraction, pass a JSON Schema and a prompt:

```python theme={null}
@app.route("/extract-product", methods=["POST"])
def extract_product():
    url = request.json["url"]
    result = spidra.scrape(
        url,
        prompt="Extract product details",
        schema={
            "type": "object",
            "properties": {
                "title":     {"type": "string"},
                "price":     {"type": "number"},
                "in_stock":  {"type": "boolean"},
            },
            "required": ["title", "price", "in_stock"],
        },
        output="json",
    )
    return spidra_jsonify(result.content)
```

`spidra_jsonify()` recursively serializes Spidra dataclasses to JSON, including nested stats objects. It accepts any mix of dicts, lists, and Spidra objects.

## Searching

```python theme={null}
@app.route("/search")
def search():
    query = request.args.get("q", "")
    results = spidra.search(query, sources=["web", "news"], limit=10)
    return spidra_jsonify(results)
```

Available `sources`: `"web"`, `"news"`, `"images"`, `"videos"`, `"research"`, `"developer"`. Defaults to `["web"]`.

## Crawling

Crawl an entire site recursively and extract structured data from every page:

```python theme={null}
@app.route("/crawl", methods=["POST"])
def crawl():
    start_url = request.json["url"]
    job = spidra.crawl(
        start_url,
        crawl_instruction="Follow blog post links only",
        transform_instruction="Extract title, author, publish date, and summary",
        max_pages=30,
    )
    return spidra_jsonify([{"url": p.url, "data": p.data} for p in job.result])
```

For large crawls, submit without waiting and poll:

```python theme={null}
queued = spidra.start_crawl(start_url, crawl_instruction="...", max_pages=50)
status = spidra.get_crawl(queued.job_id)
```

## Mapping

Discover all URLs on a domain without scraping page bodies:

```python theme={null}
result = spidra.map("https://docs.example.com", search="api", limit=100)
for link in result.links:
    print(link.url, link.title)
```

## Batch scraping

Submit up to 50 URLs in one request and process them in parallel:

```python theme={null}
@app.route("/batch", methods=["POST"])
def batch():
    urls = request.json["urls"]
    batch = spidra.batch_scrape(
        urls,
        prompt="Extract the page title and meta description",
        output="json",
    )
    items = [{"url": i.url, "result": i.result} for i in batch.items if i.status == "completed"]
    return spidra_jsonify(items)
```

## Parsing documents

Parse PDFs, Word, Excel, and other documents into Markdown or structured JSON:

```python theme={null}
doc = spidra.parse("report.pdf", prompt="Invoice number, total amount, and due date")
print(doc.markdown)
```

Pass a URL to fetch and parse remotely:

```python theme={null}
paper = spidra.parse("https://arxiv.org/pdf/1706.03762", max_pages=5)
```

## Async Flask views

Flask 2.0+ supports native `async def` views. Access the async client via `spidra.async_client`:

```python theme={null}
@app.route("/async-scrape")
async def async_scrape():
    result = await spidra.async_client.scrape("https://example.com")
    return spidra_jsonify(result)
```

## Webhooks

Spidra sends signed webhook events (e.g. `crawl.page`, `crawl.completed`) to your endpoint. Use the `@spidra_webhook_required` decorator to verify HMAC-SHA256 signatures automatically:

```python theme={null}
from flask import request
from spidra_flask import spidra_webhook_required

@app.route("/webhooks/spidra", methods=["POST"])
@spidra_webhook_required
def handle_webhook():
    event = request.json
    print(f"Event: {event['event']}")
    return {"status": "ok"}
```

The decorator reads `SPIDRA_WEBHOOK_SECRET` from config, verifies the `X-Spidra-Signature` header against the raw body, and returns `401` if the signature is invalid.

### Event listeners

Register handlers that fire automatically when a verified webhook arrives. Pair this with the Blueprint's built-in `/spidra/webhooks` endpoint:

```python theme={null}
@spidra.on("crawl.completed")
def on_crawl_done(data):
    job_id = data.get("jobId")
    print(f"Crawl {job_id} finished")

@spidra.on("crawl.page")
def on_page(data):
    print(f"Page crawled: {data['page']['url']}")
```

Dispatch manually from your own endpoint:

```python theme={null}
@app.route("/hooks/spidra", methods=["POST"])
@spidra_webhook_required
def my_webhook():
    spidra.dispatch_event(request.json)
    return {"ok": True}
```

## Pre-built Blueprint

Register a Blueprint that exposes a complete Spidra REST API under `/spidra`:

```python theme={null}
from spidra_flask import spidra_bp

app.register_blueprint(spidra_bp)
```

Or with a custom prefix:

```python theme={null}
from spidra_flask import create_spidra_blueprint

bp = create_spidra_blueprint(url_prefix="/api/v1/scraper")
app.register_blueprint(bp)
```

The Blueprint exposes:

| Method | Endpoint | Description |
| - | - | - |
| `POST` | `/spidra/scrape` | Scrape a page (`url`, `prompt`, `schema`, `renderJs`, `useProxy`) |
| `GET`/`POST` | `/spidra/search` | Web search (`q`, `limit`, `sources`) |
| `POST` | `/spidra/map` | Map domain links (`url`, `limit`, `includeSubdomains`) |
| `POST` | `/spidra/crawl` | Start or execute crawl (`url`, `maxPages`, `crawlInstruction`, `async`) |
| `GET` | `/spidra/crawl/<id>` | Crawl status and results |
| `GET` | `/spidra/crawl/<id>/report` | Crawl skip report |
| `DELETE` | `/spidra/crawl/<id>` | Cancel crawl |
| `POST` | `/spidra/crawl/<id>/retry/<page_id>` | Retry one page's extraction |
| `POST` | `/spidra/batch` | Batch scrape (`urls`, `prompt`, `async`) |
| `GET` | `/spidra/batch/<id>` | Batch status |
| `DELETE` | `/spidra/batch/<id>` | Cancel batch |
| `POST` | `/spidra/batch/<id>/retry` | Retry failed items |
| `POST` | `/spidra/parse` | Parse document |
| `POST` | `/spidra/parse/estimate` | Estimate parse credits |
| `GET` | `/spidra/parse/<id>` | Parse result |
| `DELETE` | `/spidra/parse/files/<id>` | Delete uploaded file |
| `GET` | `/spidra/templates` | List templates |
| `GET` | `/spidra/templates/<id>` | Template details |
| `POST` | `/spidra/templates/<id>/run` | Run template |
| `GET`/`POST` | `/spidra/presets` | List or create presets |
| `GET`/`POST` | `/spidra/monitors` | List or create monitors |
| `POST` | `/spidra/monitors/<id>/check` | Trigger monitor check |
| `GET` | `/spidra/account` | Account and credit info |
| `GET` | `/spidra/usage` | Usage history |
| `POST` | `/spidra/webhooks` | HMAC-verified webhook receiver |

## Flask CLI

`spidra-flask` adds a `flask spidra` CLI group:

```bash theme={null}
# Account and credits
flask spidra account

# Usage breakdown
flask spidra usage --range 7d

# Scrape a URL
flask spidra scrape https://news.ycombinator.com --format markdown --prompt "Top 3 headlines"

# Web search
flask spidra search "best Python web frameworks" --limit 5

# Map a domain
flask spidra map https://example.com --limit 20

# Crawl a site
flask spidra crawl https://example.com --max-pages 10
```

## Error handling

All Spidra exceptions inherit from `SpidraError` and are re-exported from `spidra_flask`:

```python theme={null}
from spidra_flask import (
    Spidra,
    SpidraError,
    SpidraAuthenticationError,
    SpidraPaymentRequiredError,
    SpidraRateLimitError,
)

@app.errorhandler(SpidraAuthenticationError)
def handle_auth(e):
    return {"error": "Invalid API key"}, 401

@app.errorhandler(SpidraPaymentRequiredError)
def handle_payment(e):
    return {"error": "Spidra credits exhausted"}, 402

@app.errorhandler(SpidraRateLimitError)
def handle_rate_limit(e):
    return {"error": "Rate limited", "retry_after": getattr(e, "retry_after", 5)}, 429

@app.errorhandler(SpidraError)
def handle_spidra_error(e):
    return {"error": str(e)}, 500
```

See the [Python SDK](https://docs.spidra.io/sdks/python.md#error-handling) for the full exception hierarchy.

> [card] [FastAPI](https://docs.spidra.io/sdks/fastapi.md)
>
> Native async-first integration with dependency injection, Pydantic models, and OpenAPI docs.

> [card] [Django](https://docs.spidra.io/sdks/django.md)
>
> Full Django integration with management commands, Django Signals, and built-in URL routing.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.