> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Spidra MCP Server

> Use Spidra's API through the Model Context Protocol.

A [Model Context Protocol](https://modelcontextprotocol.io) (MCP) server that gives AI assistants direct access to Spidra: scraping, batch processing, and crawling the web, all through natural language instead of code. The server is open source and available on [GitHub](https://github.com/spidra-io/spidra-mcp-server).

## Features

* Scrape any page into clean markdown or structured JSON
* Batch-process up to 50 URLs in parallel, each with its own independent result
* Crawl entire sites by describing which links to follow in plain English
* Run browser actions before scraping: click, type, scroll, or loop over elements on the page
* Route through residential proxies for geo-restricted or bot-protected sites
* Streamable HTTP transport, so the server can also run as a network service for tools like n8n
* Automatic retries for transient failures, with typed, actionable error messages otherwise

## Installation

Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**. Keys start with `spd_`.

<Note>
  Spidra's MCP server runs locally today; there is no hosted, keyless endpoint yet. Every setup below uses your own API key.
</Note>

### Running with npx

```bash theme={null}
env SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcp
```

### Manual installation

```bash theme={null}
npm install -g spidra-mcp
```

### Running on Claude Code

```bash theme={null}
claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcp
```

Start a new session, then run `/mcp` to confirm the connection shows as active.

### Running on Cursor

1. Open Cursor Settings
2. Go to **Features** > **MCP Servers**
3. Click **+ Add new global MCP server**
4. Enter the following, replacing the placeholder key:

```json theme={null}
{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "YOUR-API-KEY"
      }
    }
  }
}
```

You can also put this in a `.cursor/mcp.json` file inside a single project if you only want Spidra available there.

### Running on Windsurf

Add this to `~/.codeium/windsurf/mcp_config.json`:

```json theme={null}
{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}
```

### Running on VS Code

Press `Ctrl + Shift + P` (or `Cmd + Shift + P` on Mac), type `Preferences: Open User Settings (JSON)`, and add:

```json theme={null}
{
  "mcp": {
    "servers": {
      "spidra": {
        "command": "npx",
        "args": ["-y", "spidra-mcp"],
        "env": {
          "SPIDRA_API_KEY": "YOUR_API_KEY"
        }
      }
    }
  }
}
```

To share the setup with your team instead, put the same `servers` block in a `.vscode/mcp.json` file in your repository, and use a `promptString` input for the key so it never gets committed.

### Running on Claude Desktop

Add this to the Claude config file (**Settings** > **Developer** > **Edit Config**):

```json theme={null}
{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "YOUR_API_KEY_HERE"
      }
    }
  }
}
```

Quit and reopen Claude Desktop afterward. If you see a `spawn npx ENOENT` error, Node.js is not installed or not on your system PATH; install it from [nodejs.org](https://nodejs.org) and fully restart Claude Desktop.

### Running with streamable HTTP mode

To run the server over HTTP instead of the default stdio transport:

```bash theme={null}
env HTTP_STREAMABLE_SERVER=true SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcp
```

Use the URL `http://localhost:3000/mcp`.

### Running on n8n

n8n's **AI Agent** node can call Spidra through its **MCP Client Tool** node, connecting over the streamable HTTP transport above.

1. Start the server in HTTP mode, as shown just above
2. In your n8n workflow, add an **AI Agent** node
3. In its configuration, add a new **Tool** and choose **MCP Client Tool**
4. Set the **Endpoint** to `http://localhost:3000/mcp`
5. Set **Server Transport** to **HTTP Streamable**
6. For **Tools to include**, choose **All** to expose every Spidra tool, or pick specific ones

If n8n is running somewhere else, for example inside Docker, run the MCP server on a host and port n8n can actually reach, and adjust the endpoint to match.

## Configuration

### Environment variables

* `SPIDRA_API_KEY`: your Spidra API key. Required, unless every request supplies its own key over HTTP (see below).
* `SPIDRA_API_URL` (optional): override the base API URL, for staging or self-hosted setups.
* `HTTP_STREAMABLE_SERVER` (optional): set to `true` to run the streamable HTTP transport instead of stdio.
* `PORT` / `HOST` (optional): bind address for the HTTP transport. Defaults are `3000` and `localhost`.

On the HTTP transport, the API key can also be sent per request using an `X-Spidra-API-Key` header, or an `Authorization: Bearer` header. This is useful when a single running server instance needs to serve more than one user.

## Available Tools

The server exposes 12 tools. The table below is a quick reference; each tool is described in more detail after it.

| Tool                         | What it does                                               |
| ---------------------------- | ---------------------------------------------------------- |
| `spidra_scrape`              | Scrape 1 to 3 URLs and wait for the extracted result       |
| `spidra_check_scrape_status` | Check a scrape that outlived its wait window               |
| `spidra_batch_scrape`        | Scrape 2 to 50 URLs in parallel, each with its own result  |
| `spidra_check_batch_status`  | Progress and per-URL results for a batch                   |
| `spidra_cancel_batch`        | Cancel a batch that is no longer needed                    |
| `spidra_crawl`               | Discover and process pages starting from one URL           |
| `spidra_check_crawl_status`  | Progress, then full results, for a crawl                   |
| `spidra_crawl_pages`         | Per-page results with raw HTML and markdown download links |
| `spidra_crawl_extract`       | Ask a new question of an already-completed crawl           |
| `spidra_cancel_crawl`        | Cancel a crawl that is no longer needed                    |
| `spidra_scrape_logs`         | Look up past jobs and their outputs                        |
| `spidra_usage`               | Check credit and request usage                             |

### 1. Scrape Tool (`spidra_scrape`)

Scrapes 1 to 3 URLs and extracts content with AI, waiting for the result before returning it. With more than one URL, their content is merged and the AI produces one combined answer across all of them, which makes this the right tool for comparing pages rather than getting separate results for each.

```json theme={null}
{
  "name": "spidra_scrape",
  "arguments": {
    "urls": ["https://example.com/product"],
    "prompt": "Extract the product name, price, and description",
    "output": "json",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    }
  }
}
```

**Returns:** the extracted `content`, per-page raw data, any `screenshots`, and token `stats`.

### 2. Check Scrape Status (`spidra_check_scrape_status`)

Checks a scrape job by ID. Only needed when a scrape outlives its wait window; the timeout error includes the job ID to use here.

```json theme={null}
{
  "name": "spidra_check_scrape_status",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}
```

### 3. Batch Scrape Tool (`spidra_batch_scrape`)

Submits 2 to 50 URLs processed in parallel with the same prompt or schema. Unlike `spidra_scrape`, every URL is handled independently and gets its own result. Returns immediately with a `batchId` for polling.

```json theme={null}
{
  "name": "spidra_batch_scrape",
  "arguments": {
    "urls": ["https://shop.example.com/product/1", "https://shop.example.com/product/2"],
    "prompt": "Extract the product name, price, and star rating",
    "output": "json"
  }
}
```

**Returns:** `{ "batchId": "...", "total": 2 }`.

### 4. Check Batch Status (`spidra_check_batch_status`)

Returns batch progress (`completedCount`, `failedCount`) and per-URL results for every finished item.

```json theme={null}
{
  "name": "spidra_check_batch_status",
  "arguments": { "batchId": "batch-uuid-here" }
}
```

### 5. Cancel Batch (`spidra_cancel_batch`)

Cancels a running batch. Finished items keep their results; credits for the unprocessed portion are refunded.

```json theme={null}
{
  "name": "spidra_cancel_batch",
  "arguments": { "batchId": "batch-uuid-here" }
}
```

### 6. Crawl Tool (`spidra_crawl`)

Crawls a website from one starting URL, following links according to a plain-English instruction, and optionally extracting structured data from every page. Returns immediately with a `jobId`.

```json theme={null}
{
  "name": "spidra_crawl",
  "arguments": {
    "baseUrl": "https://example.com/blog",
    "crawlInstruction": "Follow blog post links only, skip tag and category pages",
    "transformInstruction": "Extract the title, author, and publish date",
    "maxPages": 10
  }
}
```

**Returns:** `{ "jobId": "..." }`.

### 7. Check Crawl Status (`spidra_check_crawl_status`)

Returns crawl progress while running, and every page's extracted data once completed.

```json theme={null}
{
  "name": "spidra_check_crawl_status",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}
```

### 8. Crawl Pages (`spidra_crawl_pages`)

Returns per-page results with signed download URLs for each page's raw HTML and markdown. The links expire after 1 hour.

```json theme={null}
{
  "name": "spidra_crawl_pages",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}
```

### 9. Crawl Extract (`spidra_crawl_extract`)

Runs a new extraction instruction over an already-completed crawl without fetching any pages again. Only AI token credits are charged, so this is the cheap way to ask a second question of a site you already crawled.

```json theme={null}
{
  "name": "spidra_crawl_extract",
  "arguments": {
    "jobId": "550e8400-e29b-41d4-a716-446655440000",
    "transformInstruction": "List the main topics each post covers"
  }
}
```

### 10. Cancel Crawl (`spidra_cancel_crawl`)

Cancels a running crawl. Pages already processed are kept and retrievable with `spidra_crawl_pages`; credits for unprocessed pages are refunded.

```json theme={null}
{
  "name": "spidra_cancel_crawl",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}
```

### 11. Scrape Logs (`spidra_scrape_logs`)

Lists past jobs: what ran, whether it succeeded, and how many credits it used. Pass a `uuid` to fetch one job's full output.

```json theme={null}
{
  "name": "spidra_scrape_logs",
  "arguments": { "status": "failed", "limit": 10 }
}
```

### 12. Usage (`spidra_usage`)

Reports request, credit, and token usage by day or week.

```json theme={null}
{
  "name": "spidra_usage",
  "arguments": { "range": "7d" }
}
```

For the full parameter tables, tool-choice guidance, and troubleshooting steps, see the [complete MCP reference](/sdks/mcp).

## Error Handling

Every API error comes back as a tool error, not a crash. The error text carries retry guidance: rate limits say how long to wait, validation errors say exactly what to fix, and permanent failures say not to retry with the same input.

```json theme={null}
{
  "content": [
    {
      "type": "text",
      "text": "Spidra API error 404 (JOB_NOT_FOUND): Scrape not found. Do not retry with the same input."
    }
  ],
  "isError": true
}
```

## Development

The server is open source at [github.com/spidra-io/spidra-mcp-server](https://github.com/spidra-io/spidra-mcp-server).

```bash theme={null}
git clone https://github.com/spidra-io/spidra-mcp-server.git
cd spidra-mcp-server
npm install

npm run build      # bundles src/index.ts to dist/index.js
npm test           # builds, then runs the test suite against a fake API
npm run typecheck  # type-checks without emitting
```

The test suite spawns the built server as a real process and talks to it over stdio JSON-RPC, the same way an MCP client would, against a fake Spidra API so no credits are spent. Pull requests are welcome. Requires Node.js 20 or newer. MIT licensed.

<CardGroup cols={2}>
  <Card title="Full MCP Reference" icon="book-open" href="/sdks/mcp">
    Parameter tables, tool-choice guidance, and troubleshooting.
  </Card>

  <Card title="SDKs Overview" icon="puzzle-piece" href="/sdks/overview">
    Prefer code over chat? Browse the official SDKs.
  </Card>
</CardGroup>