Spidra MCP Server

A Model Context Protocol (MCP) server that gives AI assistants direct access to Spidra: scraping, batch processing, and crawling the web, all through natural language instead of code. The server is open source and available on GitHub.

Features

Scrape any page into clean markdown or structured JSON
Batch-process up to 50 URLs in parallel, each with its own independent result
Crawl entire sites by describing which links to follow in plain English
Run browser actions before scraping: click, type, scroll, or loop over elements on the page
Route through residential proxies for geo-restricted or bot-protected sites
Streamable HTTP transport, so the server can also run as a network service for tools like n8n
Automatic retries for transient failures, with typed, actionable error messages otherwise

Installation

Get your API key from app.spidra.io under Settings > API Keys. Keys start with spd_.

Spidra’s MCP server runs locally today; there is no hosted, keyless endpoint yet. Every setup below uses your own API key.

Running with npx

env SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcp

Manual installation

npm install -g spidra-mcp

Running on Claude Code

claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcp

Start a new session, then run /mcp to confirm the connection shows as active.

Running on Cursor

Open Cursor Settings
Go to Features > MCP Servers
Click + Add new global MCP server
Enter the following, replacing the placeholder key:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "YOUR-API-KEY"
      }
    }
  }
}

You can also put this in a .cursor/mcp.json file inside a single project if you only want Spidra available there.

Running on Windsurf

Add this to ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Running on VS Code

Press Ctrl + Shift + P (or Cmd + Shift + P on Mac), type Preferences: Open User Settings (JSON), and add:

{
  "mcp": {
    "servers": {
      "spidra": {
        "command": "npx",
        "args": ["-y", "spidra-mcp"],
        "env": {
          "SPIDRA_API_KEY": "YOUR_API_KEY"
        }
      }
    }
  }
}

To share the setup with your team instead, put the same servers block in a .vscode/mcp.json file in your repository, and use a promptString input for the key so it never gets committed.

Running on Claude Desktop

Add this to the Claude config file (Settings > Developer > Edit Config):

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "YOUR_API_KEY_HERE"
      }
    }
  }
}

Quit and reopen Claude Desktop afterward. If you see a spawn npx ENOENT error, Node.js is not installed or not on your system PATH; install it from nodejs.org and fully restart Claude Desktop.

Running with streamable HTTP mode

To run the server over HTTP instead of the default stdio transport:

env HTTP_STREAMABLE_SERVER=true SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcp

Use the URL http://localhost:3000/mcp.

Running on n8n

n8n’s AI Agent node can call Spidra through its MCP Client Tool node, connecting over the streamable HTTP transport above.

Start the server in HTTP mode, as shown just above
In your n8n workflow, add an AI Agent node
In its configuration, add a new Tool and choose MCP Client Tool
Set the Endpoint to http://localhost:3000/mcp
Set Server Transport to HTTP Streamable
For Tools to include, choose All to expose every Spidra tool, or pick specific ones

If n8n is running somewhere else, for example inside Docker, run the MCP server on a host and port n8n can actually reach, and adjust the endpoint to match.

Configuration

Environment variables

SPIDRA_API_KEY: your Spidra API key. Required, unless every request supplies its own key over HTTP (see below).
SPIDRA_API_URL (optional): override the base API URL, for staging or self-hosted setups.
HTTP_STREAMABLE_SERVER (optional): set to true to run the streamable HTTP transport instead of stdio.
PORT / HOST (optional): bind address for the HTTP transport. Defaults are 3000 and localhost.

On the HTTP transport, the API key can also be sent per request using an X-Spidra-API-Key header, or an Authorization: Bearer header. This is useful when a single running server instance needs to serve more than one user.

Available Tools

The server exposes 12 tools. The table below is a quick reference; each tool is described in more detail after it.

Tool	What it does
`spidra_scrape`	Scrape 1 to 3 URLs and wait for the extracted result
`spidra_check_scrape_status`	Check a scrape that outlived its wait window
`spidra_batch_scrape`	Scrape 2 to 50 URLs in parallel, each with its own result
`spidra_check_batch_status`	Progress and per-URL results for a batch
`spidra_cancel_batch`	Cancel a batch that is no longer needed
`spidra_crawl`	Discover and process pages starting from one URL
`spidra_check_crawl_status`	Progress, then full results, for a crawl
`spidra_crawl_pages`	Per-page results with raw HTML and markdown download links
`spidra_crawl_extract`	Ask a new question of an already-completed crawl
`spidra_cancel_crawl`	Cancel a crawl that is no longer needed
`spidra_scrape_logs`	Look up past jobs and their outputs
`spidra_usage`	Check credit and request usage

1. Scrape Tool (`spidra_scrape`)

Scrapes 1 to 3 URLs and extracts content with AI, waiting for the result before returning it. With more than one URL, their content is merged and the AI produces one combined answer across all of them, which makes this the right tool for comparing pages rather than getting separate results for each.

{
  "name": "spidra_scrape",
  "arguments": {
    "urls": ["https://example.com/product"],
    "prompt": "Extract the product name, price, and description",
    "output": "json",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    }
  }
}

Returns: the extracted content, per-page raw data, any screenshots, and token stats.

2. Check Scrape Status (`spidra_check_scrape_status`)

Checks a scrape job by ID. Only needed when a scrape outlives its wait window; the timeout error includes the job ID to use here.

{
  "name": "spidra_check_scrape_status",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}

3. Batch Scrape Tool (`spidra_batch_scrape`)

Submits 2 to 50 URLs processed in parallel with the same prompt or schema. Unlike spidra_scrape, every URL is handled independently and gets its own result. Returns immediately with a batchId for polling.

{
  "name": "spidra_batch_scrape",
  "arguments": {
    "urls": ["https://shop.example.com/product/1", "https://shop.example.com/product/2"],
    "prompt": "Extract the product name, price, and star rating",
    "output": "json"
  }
}

Returns: { "batchId": "...", "total": 2 }.

4. Check Batch Status (`spidra_check_batch_status`)

Returns batch progress (completedCount, failedCount) and per-URL results for every finished item.

{
  "name": "spidra_check_batch_status",
  "arguments": { "batchId": "batch-uuid-here" }
}

5. Cancel Batch (`spidra_cancel_batch`)

Cancels a running batch. Finished items keep their results; credits for the unprocessed portion are refunded.

{
  "name": "spidra_cancel_batch",
  "arguments": { "batchId": "batch-uuid-here" }
}

6. Crawl Tool (`spidra_crawl`)

Crawls a website from one starting URL, following links according to a plain-English instruction, and optionally extracting structured data from every page. Returns immediately with a jobId.

{
  "name": "spidra_crawl",
  "arguments": {
    "baseUrl": "https://example.com/blog",
    "crawlInstruction": "Follow blog post links only, skip tag and category pages",
    "transformInstruction": "Extract the title, author, and publish date",
    "maxPages": 10
  }
}

Returns: { "jobId": "..." }.

7. Check Crawl Status (`spidra_check_crawl_status`)

Returns crawl progress while running, and every page’s extracted data once completed.

{
  "name": "spidra_check_crawl_status",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}

8. Crawl Pages (`spidra_crawl_pages`)

Returns per-page results with signed download URLs for each page’s raw HTML and markdown. The links expire after 1 hour.

{
  "name": "spidra_crawl_pages",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}

9. Crawl Extract (`spidra_crawl_extract`)

Runs a new extraction instruction over an already-completed crawl without fetching any pages again. Only AI token credits are charged, so this is the cheap way to ask a second question of a site you already crawled.

{
  "name": "spidra_crawl_extract",
  "arguments": {
    "jobId": "550e8400-e29b-41d4-a716-446655440000",
    "transformInstruction": "List the main topics each post covers"
  }
}

10. Cancel Crawl (`spidra_cancel_crawl`)

Cancels a running crawl. Pages already processed are kept and retrievable with spidra_crawl_pages; credits for unprocessed pages are refunded.

{
  "name": "spidra_cancel_crawl",
  "arguments": { "jobId": "550e8400-e29b-41d4-a716-446655440000" }
}

11. Scrape Logs (`spidra_scrape_logs`)

Lists past jobs: what ran, whether it succeeded, and how many credits it used. Pass a uuid to fetch one job’s full output.

{
  "name": "spidra_scrape_logs",
  "arguments": { "status": "failed", "limit": 10 }
}

12. Usage (`spidra_usage`)

Reports request, credit, and token usage by day or week.

{
  "name": "spidra_usage",
  "arguments": { "range": "7d" }
}

For the full parameter tables, tool-choice guidance, and troubleshooting steps, see the complete MCP reference.

Error Handling

Every API error comes back as a tool error, not a crash. The error text carries retry guidance: rate limits say how long to wait, validation errors say exactly what to fix, and permanent failures say not to retry with the same input.

{
  "content": [
    {
      "type": "text",
      "text": "Spidra API error 404 (JOB_NOT_FOUND): Scrape not found. Do not retry with the same input."
    }
  ],
  "isError": true
}

Development

The server is open source at github.com/spidra-io/spidra-mcp-server.

git clone https://github.com/spidra-io/spidra-mcp-server.git
cd spidra-mcp-server
npm install

npm run build      # bundles src/index.ts to dist/index.js
npm test           # builds, then runs the test suite against a fake API
npm run typecheck  # type-checks without emitting

The test suite spawns the built server as a real process and talks to it over stdio JSON-RPC, the same way an MCP client would, against a fake Spidra API so no credits are spent. Pull requests are welcome. Requires Node.js 20 or newer. MIT licensed.

Full MCP Reference

Parameter tables, tool-choice guidance, and troubleshooting.

SDKs Overview

Prefer code over chat? Browse the official SDKs.

Get Started

Core Features

Advanced

Dashboard

Billing & Usage

Integrations

Spidra MCP Server

Features

Installation

Running with npx

Manual installation

Running on Claude Code

Running on Cursor

Running on Windsurf

Running on VS Code

Running on Claude Desktop

Running with streamable HTTP mode

Running on n8n

Configuration

Environment variables

Available Tools

1. Scrape Tool (`spidra_scrape`)

2. Check Scrape Status (`spidra_check_scrape_status`)

3. Batch Scrape Tool (`spidra_batch_scrape`)

4. Check Batch Status (`spidra_check_batch_status`)

5. Cancel Batch (`spidra_cancel_batch`)

6. Crawl Tool (`spidra_crawl`)

7. Check Crawl Status (`spidra_check_crawl_status`)

8. Crawl Pages (`spidra_crawl_pages`)

9. Crawl Extract (`spidra_crawl_extract`)

10. Cancel Crawl (`spidra_cancel_crawl`)

11. Scrape Logs (`spidra_scrape_logs`)

12. Usage (`spidra_usage`)

Error Handling

Development

Full MCP Reference

SDKs Overview

​Features

​Installation

​Running with npx

​Manual installation

​Running on Claude Code

​Running on Cursor

​Running on Windsurf

​Running on VS Code

​Running on Claude Desktop

​Running with streamable HTTP mode

​Running on n8n

​Configuration

​Environment variables

​Available Tools

​1. Scrape Tool (spidra_scrape)

​2. Check Scrape Status (spidra_check_scrape_status)

​3. Batch Scrape Tool (spidra_batch_scrape)

​4. Check Batch Status (spidra_check_batch_status)

​5. Cancel Batch (spidra_cancel_batch)

​6. Crawl Tool (spidra_crawl)

​7. Check Crawl Status (spidra_check_crawl_status)

​8. Crawl Pages (spidra_crawl_pages)

​9. Crawl Extract (spidra_crawl_extract)

​10. Cancel Crawl (spidra_cancel_crawl)

​11. Scrape Logs (spidra_scrape_logs)

​12. Usage (spidra_usage)

​Error Handling

​Development

Full MCP Reference

SDKs Overview

Features

Installation

Running with npx

Manual installation

Running on Claude Code

Running on Cursor

Running on Windsurf

Running on VS Code

Running on Claude Desktop

Running with streamable HTTP mode

Running on n8n

Configuration

Environment variables

Available Tools

1. Scrape Tool (`spidra_scrape`)

2. Check Scrape Status (`spidra_check_scrape_status`)

3. Batch Scrape Tool (`spidra_batch_scrape`)

4. Check Batch Status (`spidra_check_batch_status`)

5. Cancel Batch (`spidra_cancel_batch`)

6. Crawl Tool (`spidra_crawl`)

7. Check Crawl Status (`spidra_check_crawl_status`)

8. Crawl Pages (`spidra_crawl_pages`)

9. Crawl Extract (`spidra_crawl_extract`)

10. Cancel Crawl (`spidra_cancel_crawl`)

11. Scrape Logs (`spidra_scrape_logs`)

12. Usage (`spidra_usage`)

Error Handling

Development