> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.
# MCP Server
> Connect Spidra to Claude, Cursor, and any MCP-compatible AI agent.
MCP is the standard that lets AI assistants use external tools. The [Spidra MCP server](https://github.com/spidra-io/spidra-mcp-server) gives any MCP-compatible assistant like Claude Code, Claude Desktop, Cursor, Windsurf, and VS Code the ability to scrape pages, batch-process URLs, and crawl entire sites on its own.
You describe what you want in chat and the assistant picks the right Spidra tool, runs it, and works with the extracted data directly. You never call the tools by name, and you do not write any code.
The server ships with 12 tools. Their descriptions teach the model when to use which, including how to keep your credit spend low.
## Before you start
You need three things:
1. **A Spidra API key.** Create one at [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**. Keys start with `spd_`.
2. **Node.js 20 or newer.** Check with `node --version`. The `npx` command that runs the server ships with Node.
3. **An MCP-compatible client.** Any of the assistants below works.
## Setup
Every setup below does the same thing. It tells your assistant to run `npx -y spidra-mcp` and hands the server your API key through an environment variable.
Run this one command in your terminal, replacing the placeholder with your real key:
```bash theme={null}
claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcp
```
Start a new Claude Code session, then run `/mcp` to confirm the connection shows as active.
1. Open Cursor Settings
2. Go to **Features** > **MCP Servers**
3. Click **+ Add new global MCP server**
4. Paste the following and replace the placeholder key:
```json theme={null}
{
"mcpServers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
}
}
}
```
You can also put this in a `.cursor/mcp.json` file inside a project if you only want Spidra available there.
1. Open **Settings** > **Developer** > **Edit Config**. This opens `claude_desktop_config.json`
2. Add the `spidra` entry inside `mcpServers`:
```json theme={null}
{
"mcpServers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
}
}
}
```
3. Quit and reopen Claude Desktop. The tools appear under the tools icon in the chat input.
Press `Ctrl + Shift + P` (or `Cmd + Shift + P` on Mac), type `Preferences: Open User Settings (JSON)`, and add:
```json theme={null}
{
"mcp": {
"servers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
}
}
}
}
```
To share the setup with your team instead, put the same `servers` block in a `.vscode/mcp.json` file in your repository and use a `promptString` input for the key so it never gets committed.
Add to `~/.codeium/windsurf/mcp_config.json`:
```json theme={null}
{
"mcpServers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
}
}
}
```
## Try it
Once connected, just ask for web data in normal language:
> "Scrape [https://news.ycombinator.com](https://news.ycombinator.com) and give me the top 5 stories with their points."
> "Compare the pricing pages of stripe.com and paddle.com and tell me which is cheaper for a small SaaS."
> "Here are 12 product URLs. Get me the name, price, and rating for each one as a table."
> "Crawl the first 10 pages of docs.example.com and summarize what the product does."
If the assistant answers with real data from those pages, everything is working. The assistant chooses the tool, waits or polls as needed, and answers with the extracted data.
## How to choose a tool
This guidance is embedded in the tool descriptions, so the assistant follows it on its own. It is worth knowing yourself so you can phrase requests well.
The deciding question is not how many URLs you have. It is what you want back:
* **You want one answer, and you know the URL (or 2 to 3 related URLs):** the assistant uses **`spidra_scrape`**. With several URLs, their content is merged and the AI answers once across all of them. That makes it the right tool for comparing two pricing pages or summarizing three related articles into one answer.
* **You want separate data for each URL in a list:** the assistant uses **`spidra_batch_scrape`**, even for just 2 URLs. Every URL is processed independently and returns its own result.
* **You do not know the page URLs yet:** the assistant uses **`spidra_crawl`**. It takes one starting URL and a plain-English instruction about which links to follow, and discovers the pages itself.
| Tool | Best for | Waits or polls? |
| ---------------------------- | ---------------------------------------------------------- | ------------------------------------ |
| `spidra_scrape` | One combined answer from 1 to 3 known URLs | Waits, returns the result directly |
| `spidra_check_scrape_status` | Re-checking a scrape that outlived its wait window | Instant lookup |
| `spidra_batch_scrape` | Separate results for each of 2 to 50 known URLs | Returns a `batchId`, assistant polls |
| `spidra_check_batch_status` | Progress and per-URL results for a batch | Instant lookup |
| `spidra_cancel_batch` | Stopping a batch you no longer need | Instant |
| `spidra_crawl` | Discovering and processing pages from one starting URL | Returns a `jobId`, assistant polls |
| `spidra_check_crawl_status` | Progress, then full results, for a crawl | Instant lookup |
| `spidra_crawl_pages` | Per-page results with raw HTML and markdown download links | Instant lookup |
| `spidra_crawl_extract` | Asking a new question of an already-completed crawl | Returns a new `jobId` |
| `spidra_cancel_crawl` | Stopping a crawl you no longer need | Instant |
| `spidra_scrape_logs` | Looking up past jobs and their outputs | Instant lookup |
| `spidra_usage` | Checking credit and request usage | Instant lookup |
When you need specific fields from a page, ask for JSON and describe the fields. The assistant gets back a small, focused payload instead of an entire page. Ask for full markdown only when you genuinely need the whole page.
## The tools in detail
Scrapes 1 to 3 URLs and extracts their content with AI. Waits for the result, typically 10 to 60 seconds, and returns the extracted content directly.
The important behavior to understand: with more than one URL, the pages' content is combined and the AI produces **one answer across all of them**. The raw per-page content still comes back in `pages`, but the extraction is a single merged result. Use multiple URLs to compare or synthesize across pages. For separate results per URL, the assistant should use `spidra_batch_scrape` instead, even for 2 URLs.
**Prompt example:**
> "Get the product name, price, and description from [https://example.com/product](https://example.com/product)."
**Usage example (structured extraction with a schema):**
```json theme={null}
{
"name": "spidra_scrape",
"arguments": {
"urls": ["https://example.com/product"],
"prompt": "Extract the product information",
"output": "json",
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number" },
"description": { "type": "string" }
},
"required": ["name", "price"]
}
}
}
```
**Other options:** `actions` (browser steps to run first: click, type, scroll, forEach loops), `cookies` (for pages behind a login), `useProxy` and `proxyCountry` (residential proxy routing), `screenshot`, `extractContentOnly` (strip boilerplate), and `scrapeMode: "fast"` (HTTP only, no browser, cheaper but cannot run actions).
**Returns:** the extracted `content`, per-page raw data in `pages`, any `screenshots`, and `stats` with token counts. If the wait window is exceeded, the job keeps running server-side and the error hands the assistant the job ID to check with `spidra_check_scrape_status`.
Submits 2 to 50 URLs processed in parallel with the same prompt or schema. Each URL is handled **independently and gets its own result**. Returns immediately with a `batchId`; the assistant polls `spidra_check_batch_status` every 10 to 15 seconds until the batch finishes. The tool response tells it to do exactly that.
**Prompt example:**
> "Here are 15 product URLs. Extract the name, price, and star rating from each one."
**Usage example:**
```json theme={null}
{
"name": "spidra_batch_scrape",
"arguments": {
"urls": [
"https://shop.example.com/product/1",
"https://shop.example.com/product/2"
],
"prompt": "Extract the product name, price, and star rating",
"output": "json"
}
}
```
**Returns:** `{ "batchId": "...", "total": 2 }` plus polling instructions. URLs are plain strings, not objects.
A `completed` batch can still contain individual failed items; check `failedCount` in the status. Failed items can be retried through the [batch retry endpoint](/api-reference/scraping/batch-scrape-retry) without re-running the whole batch.
Crawls a website starting from one URL, following links according to a plain-English instruction, and optionally extracting structured data from every page. Returns immediately with a `jobId`; the assistant polls `spidra_check_crawl_status`.
Two instructions control a crawl, and they do different jobs:
* `crawlInstruction` controls **which links get followed**: "Follow blog post links only, skip tag and category pages."
* `transformInstruction` controls **what gets extracted from each page**: "Extract the title, author, and publish date." Leave it out (with no schema) and each page comes back as raw markdown with no AI token cost at all.
**Prompt example:**
> "Crawl example.com/blog, follow only the article links, and get me each post's title, author, and date. Cap it at 10 pages."
**Usage example:**
```json theme={null}
{
"name": "spidra_crawl",
"arguments": {
"baseUrl": "https://example.com/blog",
"crawlInstruction": "Follow blog post links only, skip tag and category pages",
"transformInstruction": "Extract the title, author, and publish date",
"maxPages": 10
}
}
```
**Scoping options:** `maxPages` (default 5, maximum 50), `maxDepth`, `includePaths` and `excludePaths`, `allowSubdomains`, `crawlEntireDomain`, and `ignoreQueryParams`. Every crawled page costs credits, so start small. You can always crawl again, or better, re-extract.
**Returns:** `{ "jobId": "..." }` plus polling instructions.
Runs a brand new extraction instruction over a crawl that already completed, without fetching any pages again. Spidra kept the page content, so only AI token credits are charged. This is the cheap way to ask a second question of the same site.
For example: you crawled a competitor's blog extracting titles and dates. Now you want the key topics of each post too. Re-extract instead of re-crawling.
```json theme={null}
{
"name": "spidra_crawl_extract",
"arguments": {
"jobId": "550e8400-e29b-41d4-a716-446655440000",
"transformInstruction": "List the main topics each post covers"
}
}
```
**Returns:** a new `jobId`, polled like a normal crawl. The source crawl must have status `completed`.
**`spidra_check_scrape_status`** looks up a scrape by job ID. Only needed when a scrape outlived its wait window; the timeout error includes the ID.
**`spidra_check_batch_status`** returns batch progress (`completedCount`, `failedCount`) and per-URL results for every finished item.
**`spidra_check_crawl_status`** returns crawl progress (`pagesCrawled` out of `maxPages`) while running, and every page's extracted data once completed.
**`spidra_crawl_pages`** returns per-page results with signed download URLs for each page's raw HTML and markdown. The links expire after 1 hour. Works on cancelled crawls too, returning whatever finished before the cancellation.
**`spidra_cancel_batch`** and **`spidra_cancel_crawl`** stop a running job. Finished work is kept, and credits for the unprocessed portion are refunded automatically.
**`spidra_scrape_logs`** browses past jobs: what ran, whether it succeeded, and how many credits it used. Filters by status and search term; pass a `uuid` for one job's full AI output. Ask the assistant "why did my last scrape fail?" and this is what it reaches for.
**`spidra_usage`** reports request, credit, and token usage by day or week (`"7d"`, `"30d"`, or `"weekly"`). Useful before kicking off a large batch or crawl.
## How it protects your credits
Agent loops can burn credits fast if the tools let them. The server is built to prevent that:
* The tool descriptions steer the model toward the cheapest tool that answers the question, and tell it to keep `maxPages` small.
* Long-running jobs return a job ID with explicit polling instructions, so the model never resubmits a job that is still running. Timeout errors say the job is still running and to poll, not retry.
* Error messages carry retry guidance: rate limits say exactly how long to wait, validation errors list exactly what to fix, and permanent errors say not to retry at all. No expensive retry loops on requests that can never succeed.
* Cancelling unfinished work refunds unprocessed items, and the cancel tools say so.
## Output size
Large pages and big crawls can produce more text than fits in a model's context window. The server truncates individual strings above 5,000 characters and caps any single tool response at 80,000 characters, marking every truncation clearly. When the assistant needs the complete raw content of crawled pages, `spidra_crawl_pages` provides download links to the full files.
## Configuration
| Env var | Required | Description |
| ------------------------ | -------- | ---------------------------------------------------------------------------------------------------- |
| `SPIDRA_API_KEY` | Yes | Your Spidra API key, starting with `spd_` |
| `SPIDRA_API_URL` | No | Override the API base URL, for staging or self-hosted setups |
| `HTTP_STREAMABLE_SERVER` | No | Set to `true` to serve the HTTP streamable transport at `http://localhost:3000/mcp` instead of stdio |
| `PORT` / `HOST` | No | Bind address for the HTTP transport. Defaults are `3000` and `localhost` |
On the HTTP transport, the API key can also be sent per request via an `X-Spidra-API-Key` or `Authorization: Bearer` header, which is useful when one server instance serves more than one user.
## Troubleshooting
**The assistant does not see any Spidra tools.** Restart your client after adding the config. Most clients only read MCP configuration at startup. In Claude Code, run `/mcp` to check the connection status.
**"No Spidra API key configured."** The `SPIDRA_API_KEY` variable is not reaching the server. Make sure it is inside the `env` block of the server entry, not at the top level of the config file, and that the key still exists under **Settings** > **API Keys** in your dashboard.
**A scrape "timed out."** The job is still running server-side and nothing is lost. The error includes the job ID, and the assistant will fetch the result with `spidra_check_scrape_status`. Bot-protected sites can take a couple of minutes.
**Results come back empty when using a schema.** Check the schema: every field you want must be defined with a type. An object with no properties gives the AI nothing to fill in.
**`npx` cannot find the package.** Make sure you are on Node 20 or newer and that your network allows access to the npm registry.
Use Spidra as a document loader or agent tool inside LangChain pipelines.
Prefer code over chat? Browse the official SDKs.