MCP Server

You describe the result you want. Your assistant chooses the appropriate Spidra tool, runs it, waits for any background work to finish, and works with the returned data. In normal use, you do not need to call MCP tools by name or write integration code yourself. The server exposes 12 tools for scraping, crawling, job management, and usage reporting. Their descriptions also help the model choose the right tool and avoid unnecessary credit usage. The Spidra MCP server is open source on GitHub.

Before you start

You need:

A Spidra account. Sign up at app.spidra.io.
An MCP-compatible client. Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Antigravity, OpenCode, and other MCP clients can connect to Spidra.

If you use OAuth, you do not need to create an API key manually. If you prefer a static credential, create an API key under Settings → API Keys in your Spidra dashboard. Spidra API keys start with spd_.

Connect to the hosted server

For most users, the hosted MCP server is the recommended setup. There is nothing to install, run, or update locally.

https://mcp.spidra.io/mcp

If your client supports OAuth, add the endpoint and sign in to Spidra when prompted. For CI, scripting, or clients where you prefer a static credential, you can send a Spidra API key in the Authorization header instead:

Authorization: Bearer spd_YOUR_API_KEY

When you sign in through OAuth, Spidra creates or reuses a dedicated API key labeled MCP (OAuth) on your account. You can view or revoke it at any time under Settings → API Keys.

Set up your client

Run:

claude mcp add --transport http spidra https://mcp.spidra.io/mcp

Start a new Claude Code session, then run /mcp to complete authentication or confirm that Spidra is connected.If you want to use an API key instead:

claude mcp add --transport http spidra https://mcp.spidra.io/mcp \
  --header "Authorization: Bearer spd_YOUR_API_KEY"

Start a new session and run /mcp to confirm the connection.

Open Customize → Connectors, click +, then select Add custom connector.Enter:

https://mcp.spidra.io/mcp

Click Add, then connect to Spidra when Claude prompts you to authenticate.If you need to use a static API key instead of OAuth, use the local setup below.

Add Spidra to your MCP configuration.For a project-specific setup, create:

.cursor/mcp.json

For a global setup, use:

~/.cursor/mcp.json

Add:

{
  "mcpServers": {
    "spidra": {
      "url": "https://mcp.spidra.io/mcp"
    }
  }
}

Cursor will start the authentication flow when Spidra requires it.If you want to use an API key instead, set SPIDRA_API_KEY in your environment and add the authorization header:

{
  "mcpServers": {
    "spidra": {
      "url": "https://mcp.spidra.io/mcp",
      "headers": {
        "Authorization": "Bearer ${env:SPIDRA_API_KEY}"
      }
    }
  }
}

Run MCP: Add Server from the Command Palette, or add Spidra directly to an mcp.json file.For a workspace-specific configuration, use:

.vscode/mcp.json

Add:

{
  "servers": {
    "spidra": {
      "type": "http",
      "url": "https://mcp.spidra.io/mcp"
    }
  }
}

VS Code will handle authentication when the server requests it.If you want to use an API key instead:

{
  "servers": {
    "spidra": {
      "type": "http",
      "url": "https://mcp.spidra.io/mcp",
      "headers": {
        "Authorization": "Bearer spd_YOUR_API_KEY"
      }
    }
  }
}

Do not commit a real API key to a shared workspace file. For team configurations, use a VS Code input variable or another secret-management method.

Open Windsurf Settings → Cascade → MCP Servers, or edit:

~/.codeium/windsurf/mcp_config.json

Add:

{
  "mcpServers": {
    "spidra": {
      "serverUrl": "https://mcp.spidra.io/mcp"
    }
  }
}

Windsurf supports OAuth for remote MCP servers and will prompt you when authentication is required.If you want to use an API key instead:

{
  "mcpServers": {
    "spidra": {
      "serverUrl": "https://mcp.spidra.io/mcp",
      "headers": {
        "Authorization": "Bearer spd_YOUR_API_KEY"
      }
    }
  }
}

Open Settings → Customizations → Installed MCP Servers, or edit the raw MCP configuration.Global configuration:

~/.gemini/config/mcp_config.json

Workspace configuration:

.agents/mcp_config.json

Add:

{
  "mcpServers": {
    "spidra": {
      "serverUrl": "https://mcp.spidra.io/mcp"
    }
  }
}

Antigravity handles OAuth automatically for compatible remote MCP servers.If you want to use an API key instead:

{
  "mcpServers": {
    "spidra": {
      "serverUrl": "https://mcp.spidra.io/mcp",
      "headers": {
        "Authorization": "Bearer spd_YOUR_API_KEY"
      }
    }
  }
}

For OpenCode v2, define Spidra under mcp.servers in your opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "spidra": {
        "type": "remote",
        "url": "https://mcp.spidra.io/mcp"
      }
    }
  }
}

OAuth is enabled for remote servers by default.When authentication is required, run /connect in OpenCode and choose Spidra under Services.If you want to use an API key instead, set the key in your environment:

export SPIDRA_API_KEY=spd_YOUR_API_KEY

Then disable OAuth for this server and pass the key in the request headers:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "spidra": {
        "type": "remote",
        "url": "https://mcp.spidra.io/mcp",
        "oauth": false,
        "headers": {
          "Authorization": "Bearer {env:SPIDRA_API_KEY}"
        }
      }
    }
  }
}

Run the server locally

You can also run the Spidra MCP server on your own machine over stdio. Use the local setup when:

your client does not support remote MCP servers
you want the MCP process to run locally
you need to connect to a staging or custom Spidra API through SPIDRA_API_URL

Local setups require Node.js 20 or newer. Each configuration below starts npx -y spidra-mcp and passes your Spidra API key to the process through SPIDRA_API_KEY.

Run:

claude mcp add spidra \
  -e SPIDRA_API_KEY=spd_YOUR_API_KEY \
  -- npx -y spidra-mcp

Start a new Claude Code session and run /mcp to confirm that Spidra is connected.

Open Settings → Developer → Edit Config and add Spidra under mcpServers:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "spd_YOUR_API_KEY"
      }
    }
  }
}

Quit and reopen Claude Desktop after saving the configuration.

Add this to .cursor/mcp.json for the current project or ~/.cursor/mcp.json for a global configuration:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "spd_YOUR_API_KEY"
      }
    }
  }
}

Add this to .vscode/mcp.json, or use MCP: Open User Configuration for a user-level configuration:

{
  "servers": {
    "spidra": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "spd_YOUR_API_KEY"
      }
    }
  }
}

Do not commit a real API key to a shared workspace file.

Add this to:

~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "spd_YOUR_API_KEY"
      }
    }
  }
}

Add this to ~/.gemini/config/mcp_config.json for a global setup or .agents/mcp_config.json for the current workspace:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "spd_YOUR_API_KEY"
      }
    }
  }
}

Set your Spidra API key in the environment:

export SPIDRA_API_KEY=spd_YOUR_API_KEY

Then add:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "spidra": {
        "type": "local",
        "command": ["npx", "-y", "spidra-mcp"],
        "environment": {
          "SPIDRA_API_KEY": "{env:SPIDRA_API_KEY}"
        }
      }
    }
  }
}

Try it

Once Spidra is connected, ask for web data in normal language.

“Scrape https://news.ycombinator.com and give me the top five stories with their points.”

“Compare the pricing pages for stripe.com and paddle.com and tell me which is cheaper for a small SaaS.”

“Here are 12 product URLs. Get the name, price, and rating from each one and return a table.”

“Crawl the first 10 pages of docs.example.com and summarize what the product does.”

If the assistant returns data from the requested pages, the connection is working. The assistant handles tool selection, polling, and result retrieval automatically.

Choose the right tool

You usually do not need to select a tool yourself. This guidance is built into the tool descriptions so the model can choose appropriately. The main decision is what result you want back, not simply how many URLs you have.

Use spidra_scrape when you know the page URL and want one combined answer. It accepts 1 to 3 URLs. When you provide multiple URLs, Spidra combines their content into a single extraction.
Use spidra_batch_scrape when you want a separate result for every URL in a list. It accepts 2 to 50 URLs and processes each one independently.
Use spidra_crawl when you know where to start but do not already know every page URL. Spidra discovers pages by following links according to your instructions.

Tool	Best for	Behavior
`spidra_scrape`	One combined result from 1 to 3 known URLs	Waits for the result
`spidra_check_scrape_status`	Checking a scrape that exceeded its initial wait window	Instant lookup
`spidra_batch_scrape`	Separate results for 2 to 50 known URLs	Returns a `batchId` for polling
`spidra_check_batch_status`	Batch progress and completed per-URL results	Instant lookup
`spidra_cancel_batch`	Stopping an active batch	Instant
`spidra_crawl`	Discovering and processing pages from one starting URL	Returns a `jobId` for polling
`spidra_check_crawl_status`	Crawl progress and completed results	Instant lookup
`spidra_crawl_pages`	Individual crawl pages, raw HTML, and Markdown downloads	Instant lookup
`spidra_crawl_extract`	Asking a new question of an existing completed crawl	Returns a new `jobId`
`spidra_cancel_crawl`	Stopping an active crawl	Instant
`spidra_scrape_logs`	Looking up previous jobs and outputs	Instant lookup
`spidra_usage`	Checking request, credit, and token usage	Instant lookup

Tool reference

spidra_scrape: scrape 1 to 3 URLs and wait for the result

Scrapes 1 to 3 URLs and waits for the extraction result, typically for up to 10 to 60 seconds.With one URL, the behavior is straightforward: Spidra extracts the requested information from that page.With multiple URLs, Spidra combines the page content and produces one extraction across all of them. That makes spidra_scrape useful for comparison and synthesis, for example:

“Compare these two pricing pages and summarize the differences.”

If you need one independent result for every URL, use spidra_batch_scrape instead.Example request:

{
  "name": "spidra_scrape",
  "arguments": {
    "urls": ["https://example.com/product"],
    "prompt": "Extract the product information",
    "output": "json",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    }
  }
}

Useful options include:

actions for browser interactions such as clicking, typing, scrolling, and looping over elements
cookies for authenticated pages
useProxy and proxyCountry for residential proxy routing
screenshot to capture page screenshots
extractContentOnly to remove page boilerplate
scrapeMode: "fast" for HTTP-only scraping without browser actions

Returns: extracted content, raw per-page data in pages, optional screenshots, and token statistics in stats.If the scrape takes longer than the initial wait window, the job continues running. The response includes its job ID so the assistant can retrieve it with spidra_check_scrape_status instead of submitting the scrape again.

spidra_batch_scrape: separate results for 2 to 50 URLs

Processes 2 to 50 URLs in parallel using the same prompt or schema. Each URL is handled independently and receives its own result. Use it for requests such as:

“Extract the name, price, and rating from each of these 15 product pages.”

The tool returns immediately with a batchId. The assistant then checks spidra_check_batch_status until processing finishes.Example request:

{
  "name": "spidra_batch_scrape",
  "arguments": {
    "urls": [
      "https://shop.example.com/product/1",
      "https://shop.example.com/product/2"
    ],
    "prompt": "Extract the product name, price, and star rating",
    "output": "json"
  }
}

Returns: { "batchId": "...", "total": 2 }. The URLs are plain strings.A batch with a completed status can still contain individual failed items. Check failedCount and the per-URL results before treating the entire batch as successful. Failed items can be retried through the batch retry endpoint without rerunning successful items.

spidra_crawl: discover and process pages from one starting URL

Starts from one URL and discovers additional pages by following links according to a plain-English instruction. Returns a jobId immediately. The assistant checks spidra_check_crawl_status until the crawl finishes.Two instructions control different parts of the crawl:

crawlInstruction tells Spidra which links to follow.
transformInstruction tells Spidra what to extract from each page.

For example, crawlInstruction: “Follow blog post links only. Skip tag and category pages.” And transformInstruction: “Extract the title, author, and publish date.”If you omit transformInstruction and do not provide a schema, each crawled page is returned as raw Markdown without AI extraction token cost.Example request:

{
  "name": "spidra_crawl",
  "arguments": {
    "baseUrl": "https://example.com/blog",
    "crawlInstruction": "Follow blog post links only, skip tag and category pages",
    "transformInstruction": "Extract the title, author, and publish date",
    "maxPages": 10
  }
}

Useful crawl controls include maxPages (default 5, maximum 50), maxDepth, includePaths, excludePaths, allowSubdomains, crawlEntireDomain, and ignoreQueryParams. Every processed page consumes credits, so start with a small maxPages value when exploring a new site.Returns: { "jobId": "..." }.

spidra_crawl_extract: ask a new question of a completed crawl

Runs a new extraction instruction against a crawl that has already completed. It does not fetch the pages again.Use it when you already have the crawl content but want to ask a different question. For example, suppose you crawled a competitor’s blog and extracted titles and dates. You can later ask for the main topics of each article without crawling the site again.

{
  "name": "spidra_crawl_extract",
  "arguments": {
    "jobId": "550e8400-e29b-41d4-a716-446655440000",
    "transformInstruction": "List the main topics each post covers"
  }
}

Only AI extraction credits are charged because the pages are not fetched again. The source crawl must have a completed status.Returns: a new jobId, which is checked like a standard crawl.

Status, pages, and cancel tools

spidra_check_scrape_status checks a scrape by job ID. Use it only when spidra_scrape exceeded its initial wait window — the timeout response includes the job ID to check.spidra_check_batch_status returns batch progress, including completedCount, failedCount, and results for completed URLs.spidra_check_crawl_status returns crawl progress while the crawl is running and the extracted page results when it completes.spidra_crawl_pages returns individual crawl-page results and signed download URLs for each page’s raw HTML and Markdown. Download links expire after one hour. The tool also works with cancelled crawls and returns pages that finished before cancellation.spidra_cancel_batch stops a running batch. Completed items keep their results, and credits for unprocessed items are refunded.spidra_cancel_crawl stops a running crawl. Pages already processed remain available, and credits for unprocessed pages are refunded.

spidra_scrape_logs and spidra_usage: account visibility

spidra_scrape_logs looks up previous Spidra jobs, including their status and credit usage. You can filter by status or search term, or pass a uuid to retrieve a specific job and its full output. For example, ask “Why did my last scrape fail?” and the assistant can use this tool to inspect the job rather than guessing.spidra_usage reports request, credit, and token usage. Supported ranges include 7d, 30d, and weekly. This is useful before starting a large batch or crawl.

Credit-efficient behavior

MCP agents can repeat actions if the server does not give them enough context about what is already running. Spidra’s MCP tools are designed to reduce unnecessary work and duplicate credit usage.

Tool descriptions guide the model toward the most appropriate operation for each request.
Crawl guidance encourages smaller maxPages values when a large crawl is not necessary.
Long-running operations return job IDs and explicit polling instructions instead of encouraging the model to submit the same job again.
Timeout responses tell the assistant that the existing job is still running and should be checked, not restarted.
Rate-limit errors tell the assistant how long to wait before retrying.
Validation errors identify what needs to change.
Permanent errors tell the assistant when repeating the same request will not help.
Cancelling a batch or crawl refunds credits for work that has not yet been processed.

Output limits

Large pages and crawls can return more content than an AI model can use effectively in one tool response. To keep responses manageable:

individual strings longer than 5,000 characters are truncated
a single MCP tool response is capped at 80,000 characters
truncated values are marked so the assistant knows content was omitted

For complete raw crawl content, use spidra_crawl_pages. It returns signed download links to the full HTML and Markdown files.

Configuration

These environment variables apply when you run the MCP server locally or host your own instance. They are not required when you use the hosted endpoint with OAuth.

Variable	Required	Description
`SPIDRA_API_KEY`	Usually	Your Spidra API key. Required for local stdio unless authentication is supplied another way.
`SPIDRA_API_URL`	No	Overrides the Spidra API base URL for staging or custom environments.
`HTTP_STREAMABLE_SERVER`	No	Set to `true` to run the Streamable HTTP transport instead of stdio.
`PORT`	No	Port for the HTTP server. Defaults to `3000`.
`HOST`	No	Host interface for the HTTP server. Defaults to `localhost`.

For HTTP connections, an API key can also be supplied per request:

Authorization: Bearer spd_YOUR_API_KEY

This is useful when one running server needs to serve multiple users with different Spidra accounts. If a request contains both an API key header and an authenticated OAuth session, the API key in the request takes precedence.

Troubleshooting

The assistant does not see any Spidra tools. Check that the MCP server appears as connected in your client’s MCP settings. If you edited a configuration file manually, reload the MCP server or restart the client. In Claude Code, run /mcp to inspect the connection. “No Spidra API key configured.” For a local stdio setup, make sure SPIDRA_API_KEY is inside the server’s environment configuration, for example:

{
  "env": {
    "SPIDRA_API_KEY": "spd_YOUR_API_KEY"
  }
}

For the hosted server with API-key authentication, confirm that the request contains Authorization: Bearer spd_YOUR_API_KEY. Also confirm that the key still exists under Settings → API Keys in your Spidra dashboard. The hosted server returns a 401. The API key is missing, malformed, or no longer valid. Check that the header name is exactly Authorization, the value starts with Bearer, the key starts with spd_, and the key has not been revoked. If necessary, create a new key from the Spidra dashboard. OAuth does not start. Confirm that your client supports OAuth for remote MCP servers. If the client does not support OAuth, use an API key or run the MCP server locally. If authentication started but failed, begin a fresh connection rather than reusing an old authorization URL. OAuth reports an invalid or expired transaction. The authorization request may have expired or been interrupted — OAuth transactions expire after 10 minutes. Remove and reconnect the Spidra server, then complete the new authentication flow promptly. If the problem continues, use the API-key setup for that client and contact Spidra support with the error details. A scrape “timed out.” This does not mean the job failed. The job continues running on Spidra, and the timeout response includes its job ID so the assistant can retrieve it with spidra_check_scrape_status. Do not submit the same scrape again simply because the initial wait window expired. Bot-protected sites can take longer to process. Structured results are empty. Check the schema you provided. Every field you expect Spidra to return must be defined with a type:

{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    }
  }
}

An object schema with no defined properties gives the extraction model no fields to populate. npx cannot start the server. This only affects local setups. Confirm that node --version reports Node.js 20 or newer, and that your network can reach the npm registry. The hosted MCP server does not use npx and is unaffected by local Node.js issues.

Overview

Official SDKs

AI Frameworks

Before you start

Connect to the hosted server

Set up your client

Run the server locally

Try it

Choose the right tool

Tool reference

Credit-efficient behavior

Output limits

Configuration

Troubleshooting

​Before you start

​Connect to the hosted server

​Set up your client

​Run the server locally

​Try it

​Choose the right tool

​Tool reference

​Credit-efficient behavior

​Output limits

​Configuration

​Troubleshooting

Before you start

Connect to the hosted server

Set up your client

Run the server locally

Try it

Choose the right tool

Tool reference

Credit-efficient behavior

Output limits

Configuration

Troubleshooting