Flask

The Spidra Flask integration (spidra-flask) brings the full Spidra API — scrape, batch scrape, search, crawl, map, parse, templates, presets, and monitors — natively into Flask. It follows Flask extension conventions: direct initialization or the Application Factory pattern, a pre-built Blueprint exposing REST endpoints, an event dispatcher for webhooks, and a flask spidra CLI group for terminal use.

Installation

pip install spidra-flask

Requires Python 3.9+ and Flask 2.3+.

Get your API key from app.spidra.io under API Keys. Store it in an environment variable or your Flask app config. Never hardcode it.

Configuration

Configure credentials via environment variables or directly in app.config:

# .env
SPIDRA_API_KEY="spd_live_your_api_key"
SPIDRA_WEBHOOK_SECRET="whsec_your_signing_secret"

# Optional
SPIDRA_API_URL="https://api.spidra.io/api"
SPIDRA_MAX_RETRIES=3
SPIDRA_BACKOFF_FACTOR=1.0

Or in your Flask config:

app.config["SPIDRA_API_KEY"] = "spd_live_your_api_key"
app.config["SPIDRA_WEBHOOK_SECRET"] = "whsec_your_signing_secret"

Flask app config takes precedence over environment variables.

Initialization

spidra-flask supports both styles.

Direct initialization

from flask import Flask
from spidra_flask import Spidra

app = Flask(__name__)
spidra = Spidra(app)

Application Factory pattern

from flask import Flask
from spidra_flask import Spidra

spidra = Spidra()  # no app yet

def create_app():
    app = Flask(__name__)
    app.config.from_prefixed_env()
    spidra.init_app(app)
    return app

SpidraFlask is an alias for Spidra if you prefer a more explicit name.

Scraping

spidra.scrape() extracts content from a URL — plain markdown or structured JSON — with browser rendering, CAPTCHA bypass, and optional residential proxies:

from flask import request
from spidra_flask import Spidra, spidra_jsonify

app = Flask(__name__)
spidra = Spidra(app)

@app.route("/scrape", methods=["POST"])
def scrape():
    url = request.json["url"]
    result = spidra.scrape(url, output="markdown", use_proxy=True)
    return spidra_jsonify({"content": result.content, "stats": result.stats})

For structured extraction, pass a JSON Schema and a prompt:

@app.route("/extract-product", methods=["POST"])
def extract_product():
    url = request.json["url"]
    result = spidra.scrape(
        url,
        prompt="Extract product details",
        schema={
            "type": "object",
            "properties": {
                "title":     {"type": "string"},
                "price":     {"type": "number"},
                "in_stock":  {"type": "boolean"},
            },
            "required": ["title", "price", "in_stock"],
        },
        output="json",
    )
    return spidra_jsonify(result.content)

spidra_jsonify() recursively serializes Spidra dataclasses to JSON, including nested stats objects. It accepts any mix of dicts, lists, and Spidra objects.

Searching

@app.route("/search")
def search():
    query = request.args.get("q", "")
    results = spidra.search(query, sources=["web", "news"], limit=10)
    return spidra_jsonify(results)

Available sources: "web", "news", "images", "videos", "research", "developer". Defaults to ["web"].

Crawling

Crawl an entire site recursively and extract structured data from every page:

@app.route("/crawl", methods=["POST"])
def crawl():
    start_url = request.json["url"]
    job = spidra.crawl(
        start_url,
        crawl_instruction="Follow blog post links only",
        transform_instruction="Extract title, author, publish date, and summary",
        max_pages=30,
    )
    return spidra_jsonify([{"url": p.url, "data": p.data} for p in job.result])

For large crawls, submit without waiting and poll:

queued = spidra.start_crawl(start_url, crawl_instruction="...", max_pages=50)
status = spidra.get_crawl(queued.job_id)

Mapping

Discover all URLs on a domain without scraping page bodies:

result = spidra.map("https://docs.example.com", search="api", limit=100)
for link in result.links:
    print(link.url, link.title)

Batch scraping

Submit up to 50 URLs in one request and process them in parallel:

@app.route("/batch", methods=["POST"])
def batch():
    urls = request.json["urls"]
    batch = spidra.batch_scrape(
        urls,
        prompt="Extract the page title and meta description",
        output="json",
    )
    items = [{"url": i.url, "result": i.result} for i in batch.items if i.status == "completed"]
    return spidra_jsonify(items)

Parsing documents

Parse PDFs, Word, Excel, and other documents into Markdown or structured JSON:

doc = spidra.parse("report.pdf", prompt="Invoice number, total amount, and due date")
print(doc.markdown)

Pass a URL to fetch and parse remotely:

paper = spidra.parse("https://arxiv.org/pdf/1706.03762", max_pages=5)

Async Flask views

Flask 2.0+ supports native async def views. Access the async client via spidra.async_client:

@app.route("/async-scrape")
async def async_scrape():
    result = await spidra.async_client.scrape("https://example.com")
    return spidra_jsonify(result)

Webhooks

Spidra sends signed webhook events (e.g. crawl.page, crawl.completed) to your endpoint. Use the @spidra_webhook_required decorator to verify HMAC-SHA256 signatures automatically:

from flask import request
from spidra_flask import spidra_webhook_required

@app.route("/webhooks/spidra", methods=["POST"])
@spidra_webhook_required
def handle_webhook():
    event = request.json
    print(f"Event: {event['event']}")
    return {"status": "ok"}

The decorator reads SPIDRA_WEBHOOK_SECRET from config, verifies the X-Spidra-Signature header against the raw body, and returns 401 if the signature is invalid.

Event listeners

Register handlers that fire automatically when a verified webhook arrives. Pair this with the Blueprint’s built-in /spidra/webhooks endpoint:

@spidra.on("crawl.completed")
def on_crawl_done(data):
    job_id = data.get("jobId")
    print(f"Crawl {job_id} finished")

@spidra.on("crawl.page")
def on_page(data):
    print(f"Page crawled: {data['page']['url']}")

Dispatch manually from your own endpoint:

@app.route("/hooks/spidra", methods=["POST"])
@spidra_webhook_required
def my_webhook():
    spidra.dispatch_event(request.json)
    return {"ok": True}

Pre-built Blueprint

from spidra_flask import spidra_bp

app.register_blueprint(spidra_bp)

Or with a custom prefix:

from spidra_flask import create_spidra_blueprint

bp = create_spidra_blueprint(url_prefix="/api/v1/scraper")
app.register_blueprint(bp)

The Blueprint exposes:

Method	Endpoint	Description
`POST`	`/spidra/scrape`	Scrape a page (`url`, `prompt`, `schema`, `renderJs`, `useProxy`)
`GET`/`POST`	`/spidra/search`	Web search (`q`, `limit`, `sources`)
`POST`	`/spidra/map`	Map domain links (`url`, `limit`, `includeSubdomains`)
`POST`	`/spidra/crawl`	Start or execute crawl (`url`, `maxPages`, `crawlInstruction`, `async`)
`GET`	`/spidra/crawl/<id>`	Crawl status and results
`GET`	`/spidra/crawl/<id>/report`	Crawl skip report
`DELETE`	`/spidra/crawl/<id>`	Cancel crawl
`POST`	`/spidra/crawl/<id>/retry/<page_id>`	Retry one page’s extraction
`POST`	`/spidra/batch`	Batch scrape (`urls`, `prompt`, `async`)
`GET`	`/spidra/batch/<id>`	Batch status
`DELETE`	`/spidra/batch/<id>`	Cancel batch
`POST`	`/spidra/batch/<id>/retry`	Retry failed items
`POST`	`/spidra/parse`	Parse document
`POST`	`/spidra/parse/estimate`	Estimate parse credits
`GET`	`/spidra/parse/<id>`	Parse result
`DELETE`	`/spidra/parse/files/<id>`	Delete uploaded file
`GET`	`/spidra/templates`	List templates
`GET`	`/spidra/templates/<id>`	Template details
`POST`	`/spidra/templates/<id>/run`	Run template
`GET`/`POST`	`/spidra/presets`	List or create presets
`GET`/`POST`	`/spidra/monitors`	List or create monitors
`POST`	`/spidra/monitors/<id>/check`	Trigger monitor check
`GET`	`/spidra/account`	Account and credit info
`GET`	`/spidra/usage`	Usage history
`POST`	`/spidra/webhooks`	HMAC-verified webhook receiver

Flask CLI

spidra-flask adds a flask spidra CLI group:

# Account and credits
flask spidra account

# Usage breakdown
flask spidra usage --range 7d

# Scrape a URL
flask spidra scrape https://news.ycombinator.com --format markdown --prompt "Top 3 headlines"

# Web search
flask spidra search "best Python web frameworks" --limit 5

# Map a domain
flask spidra map https://example.com --limit 20

# Crawl a site
flask spidra crawl https://example.com --max-pages 10

Error handling

All Spidra exceptions inherit from SpidraError and are re-exported from spidra_flask:

from spidra_flask import (
    Spidra,
    SpidraError,
    SpidraAuthenticationError,
    SpidraPaymentRequiredError,
    SpidraRateLimitError,
)

@app.errorhandler(SpidraAuthenticationError)
def handle_auth(e):
    return {"error": "Invalid API key"}, 401

@app.errorhandler(SpidraPaymentRequiredError)
def handle_payment(e):
    return {"error": "Spidra credits exhausted"}, 402

@app.errorhandler(SpidraRateLimitError)
def handle_rate_limit(e):
    return {"error": "Rate limited", "retry_after": getattr(e, "retry_after", 5)}, 429

@app.errorhandler(SpidraError)
def handle_spidra_error(e):
    return {"error": str(e)}, 500

See the Python SDK for the full exception hierarchy.

FastAPI

Native async-first integration with dependency injection, Pydantic models, and OpenAPI docs.

Django

Full Django integration with management commands, Django Signals, and built-in URL routing.

SDKs

AI tools

Installation

Configuration

Initialization

Direct initialization

Application Factory pattern

Scraping

Searching

Crawling

Mapping

Batch scraping

Parsing documents

Async Flask views

Webhooks

Event listeners

Pre-built Blueprint

Flask CLI

Error handling

FastAPI

Django

​Installation

​Configuration

​Initialization

​Direct initialization

​Application Factory pattern

​Scraping

​Searching

​Crawling

​Mapping

​Batch scraping

​Parsing documents

​Async Flask views

​Webhooks

​Event listeners

​Pre-built Blueprint

​Flask CLI

​Error handling

FastAPI

Django

Installation

Configuration

Initialization

Direct initialization

Application Factory pattern

Scraping

Searching

Crawling

Mapping

Batch scraping

Parsing documents

Async Flask views

Webhooks

Event listeners

Pre-built Blueprint

Flask CLI

Error handling