MCP Server

MCP is the standard that lets AI assistants use external tools. The Spidra MCP server gives any MCP-compatible assistant like Claude Code, Claude Desktop, Cursor, Windsurf, and VS Code the ability to scrape pages, batch-process URLs, and crawl entire sites on its own. You describe what you want in chat and the assistant picks the right Spidra tool, runs it, and works with the extracted data directly. You never call the tools by name, and you do not write any code. The server ships with 12 tools. Their descriptions teach the model when to use which, including how to keep your credit spend low.

Before you start

You need three things:

A Spidra API key. Create one at app.spidra.io under Settings > API Keys. Keys start with spd_.
Node.js 20 or newer. Check with node --version. The npx command that runs the server ships with Node.
An MCP-compatible client. Any of the assistants below works.

Setup

Every setup below does the same thing. It tells your assistant to run npx -y spidra-mcp and hands the server your API key through an environment variable.

Claude Code
Cursor
Claude Desktop
VS Code
Windsurf

Run this one command in your terminal, replacing the placeholder with your real key:

claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcp

Start a new Claude Code session, then run /mcp to confirm the connection shows as active.

Open Cursor Settings
Go to Features > MCP Servers
Click + Add new global MCP server
Paste the following and replace the placeholder key:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
    }
  }
}

You can also put this in a .cursor/mcp.json file inside a project if you only want Spidra available there.

Open Settings > Developer > Edit Config. This opens claude_desktop_config.json
Add the spidra entry inside mcpServers:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
    }
  }
}

Quit and reopen Claude Desktop. The tools appear under the tools icon in the chat input.

Press Ctrl + Shift + P (or Cmd + Shift + P on Mac), type Preferences: Open User Settings (JSON), and add:

{
  "mcp": {
    "servers": {
      "spidra": {
        "command": "npx",
        "args": ["-y", "spidra-mcp"],
        "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
      }
    }
  }
}

To share the setup with your team instead, put the same servers block in a .vscode/mcp.json file in your repository and use a promptString input for the key so it never gets committed.

Add to ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": { "SPIDRA_API_KEY": "spd_YOUR_API_KEY" }
    }
  }
}

Try it

Once connected, just ask for web data in normal language:

“Scrape https://news.ycombinator.com and give me the top 5 stories with their points.”

“Compare the pricing pages of stripe.com and paddle.com and tell me which is cheaper for a small SaaS.”

“Here are 12 product URLs. Get me the name, price, and rating for each one as a table.”

“Crawl the first 10 pages of docs.example.com and summarize what the product does.”

If the assistant answers with real data from those pages, everything is working. The assistant chooses the tool, waits or polls as needed, and answers with the extracted data.

How to choose a tool

This guidance is embedded in the tool descriptions, so the assistant follows it on its own. It is worth knowing yourself so you can phrase requests well. The deciding question is not how many URLs you have. It is what you want back:

You want one answer, and you know the URL (or 2 to 3 related URLs): the assistant uses spidra_scrape. With several URLs, their content is merged and the AI answers once across all of them. That makes it the right tool for comparing two pricing pages or summarizing three related articles into one answer.
You want separate data for each URL in a list: the assistant uses spidra_batch_scrape, even for just 2 URLs. Every URL is processed independently and returns its own result.
You do not know the page URLs yet: the assistant uses spidra_crawl. It takes one starting URL and a plain-English instruction about which links to follow, and discovers the pages itself.

Tool	Best for	Waits or polls?
`spidra_scrape`	One combined answer from 1 to 3 known URLs	Waits, returns the result directly
`spidra_check_scrape_status`	Re-checking a scrape that outlived its wait window	Instant lookup
`spidra_batch_scrape`	Separate results for each of 2 to 50 known URLs	Returns a `batchId`, assistant polls
`spidra_check_batch_status`	Progress and per-URL results for a batch	Instant lookup
`spidra_cancel_batch`	Stopping a batch you no longer need	Instant
`spidra_crawl`	Discovering and processing pages from one starting URL	Returns a `jobId`, assistant polls
`spidra_check_crawl_status`	Progress, then full results, for a crawl	Instant lookup
`spidra_crawl_pages`	Per-page results with raw HTML and markdown download links	Instant lookup
`spidra_crawl_extract`	Asking a new question of an already-completed crawl	Returns a new `jobId`
`spidra_cancel_crawl`	Stopping a crawl you no longer need	Instant
`spidra_scrape_logs`	Looking up past jobs and their outputs	Instant lookup
`spidra_usage`	Checking credit and request usage	Instant lookup

When you need specific fields from a page, ask for JSON and describe the fields. The assistant gets back a small, focused payload instead of an entire page. Ask for full markdown only when you genuinely need the whole page.

The tools in detail

spidra_scrape: scrape 1 to 3 URLs and wait for the result

Scrapes 1 to 3 URLs and extracts their content with AI. Waits for the result, typically 10 to 60 seconds, and returns the extracted content directly.The important behavior to understand: with more than one URL, the pages’ content is combined and the AI produces one answer across all of them. The raw per-page content still comes back in pages, but the extraction is a single merged result. Use multiple URLs to compare or synthesize across pages. For separate results per URL, the assistant should use spidra_batch_scrape instead, even for 2 URLs.Prompt example:

“Get the product name, price, and description from https://example.com/product.”

Usage example (structured extraction with a schema):

{
  "name": "spidra_scrape",
  "arguments": {
    "urls": ["https://example.com/product"],
    "prompt": "Extract the product information",
    "output": "json",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    }
  }
}

Other options: actions (browser steps to run first: click, type, scroll, forEach loops), cookies (for pages behind a login), useProxy and proxyCountry (residential proxy routing), screenshot, extractContentOnly (strip boilerplate), and scrapeMode: "fast" (HTTP only, no browser, cheaper but cannot run actions).Returns: the extracted content, per-page raw data in pages, any screenshots, and stats with token counts. If the wait window is exceeded, the job keeps running server-side and the error hands the assistant the job ID to check with spidra_check_scrape_status.

spidra_batch_scrape: separate results for 2 to 50 URLs

Submits 2 to 50 URLs processed in parallel with the same prompt or schema. Each URL is handled independently and gets its own result. Returns immediately with a batchId; the assistant polls spidra_check_batch_status every 10 to 15 seconds until the batch finishes. The tool response tells it to do exactly that.Prompt example:

“Here are 15 product URLs. Extract the name, price, and star rating from each one.”

Usage example:

{
  "name": "spidra_batch_scrape",
  "arguments": {
    "urls": [
      "https://shop.example.com/product/1",
      "https://shop.example.com/product/2"
    ],
    "prompt": "Extract the product name, price, and star rating",
    "output": "json"
  }
}

Returns: { "batchId": "...", "total": 2 } plus polling instructions. URLs are plain strings, not objects.A completed batch can still contain individual failed items; check failedCount in the status. Failed items can be retried through the batch retry endpoint without re-running the whole batch.

spidra_crawl: discover and process pages from one starting URL

Crawls a website starting from one URL, following links according to a plain-English instruction, and optionally extracting structured data from every page. Returns immediately with a jobId; the assistant polls spidra_check_crawl_status.Two instructions control a crawl, and they do different jobs:

crawlInstruction controls which links get followed: “Follow blog post links only, skip tag and category pages.”
transformInstruction controls what gets extracted from each page: “Extract the title, author, and publish date.” Leave it out (with no schema) and each page comes back as raw markdown with no AI token cost at all.

Prompt example:

“Crawl example.com/blog, follow only the article links, and get me each post’s title, author, and date. Cap it at 10 pages.”

Usage example:

{
  "name": "spidra_crawl",
  "arguments": {
    "baseUrl": "https://example.com/blog",
    "crawlInstruction": "Follow blog post links only, skip tag and category pages",
    "transformInstruction": "Extract the title, author, and publish date",
    "maxPages": 10
  }
}

Scoping options: maxPages (default 5, maximum 50), maxDepth, includePaths and excludePaths, allowSubdomains, crawlEntireDomain, and ignoreQueryParams. Every crawled page costs credits, so start small. You can always crawl again, or better, re-extract.Returns: { "jobId": "..." } plus polling instructions.

spidra_crawl_extract: ask a new question of a completed crawl

Runs a brand new extraction instruction over a crawl that already completed, without fetching any pages again. Spidra kept the page content, so only AI token credits are charged. This is the cheap way to ask a second question of the same site.For example: you crawled a competitor’s blog extracting titles and dates. Now you want the key topics of each post too. Re-extract instead of re-crawling.

{
  "name": "spidra_crawl_extract",
  "arguments": {
    "jobId": "550e8400-e29b-41d4-a716-446655440000",
    "transformInstruction": "List the main topics each post covers"
  }
}

Returns: a new jobId, polled like a normal crawl. The source crawl must have status completed.

Status, pages, and cancel tools

spidra_check_scrape_status looks up a scrape by job ID. Only needed when a scrape outlived its wait window; the timeout error includes the ID.spidra_check_batch_status returns batch progress (completedCount, failedCount) and per-URL results for every finished item.spidra_check_crawl_status returns crawl progress (pagesCrawled out of maxPages) while running, and every page’s extracted data once completed.spidra_crawl_pages returns per-page results with signed download URLs for each page’s raw HTML and markdown. The links expire after 1 hour. Works on cancelled crawls too, returning whatever finished before the cancellation.spidra_cancel_batch and spidra_cancel_crawl stop a running job. Finished work is kept, and credits for the unprocessed portion are refunded automatically.

spidra_scrape_logs and spidra_usage: account visibility

spidra_scrape_logs browses past jobs: what ran, whether it succeeded, and how many credits it used. Filters by status and search term; pass a uuid for one job’s full AI output. Ask the assistant “why did my last scrape fail?” and this is what it reaches for.spidra_usage reports request, credit, and token usage by day or week ("7d", "30d", or "weekly"). Useful before kicking off a large batch or crawl.

How it protects your credits

Agent loops can burn credits fast if the tools let them. The server is built to prevent that:

The tool descriptions steer the model toward the cheapest tool that answers the question, and tell it to keep maxPages small.
Long-running jobs return a job ID with explicit polling instructions, so the model never resubmits a job that is still running. Timeout errors say the job is still running and to poll, not retry.
Error messages carry retry guidance: rate limits say exactly how long to wait, validation errors list exactly what to fix, and permanent errors say not to retry at all. No expensive retry loops on requests that can never succeed.
Cancelling unfinished work refunds unprocessed items, and the cancel tools say so.

Output size

Large pages and big crawls can produce more text than fits in a model’s context window. The server truncates individual strings above 5,000 characters and caps any single tool response at 80,000 characters, marking every truncation clearly. When the assistant needs the complete raw content of crawled pages, spidra_crawl_pages provides download links to the full files.

Configuration

Env var	Required	Description
`SPIDRA_API_KEY`	Yes	Your Spidra API key, starting with `spd_`
`SPIDRA_API_URL`	No	Override the API base URL, for staging or self-hosted setups
`HTTP_STREAMABLE_SERVER`	No	Set to `true` to serve the HTTP streamable transport at `http://localhost:3000/mcp` instead of stdio
`PORT` / `HOST`	No	Bind address for the HTTP transport. Defaults are `3000` and `localhost`

On the HTTP transport, the API key can also be sent per request via an X-Spidra-API-Key or Authorization: Bearer header, which is useful when one server instance serves more than one user.

Troubleshooting

The assistant does not see any Spidra tools. Restart your client after adding the config. Most clients only read MCP configuration at startup. In Claude Code, run /mcp to check the connection status. “No Spidra API key configured.” The SPIDRA_API_KEY variable is not reaching the server. Make sure it is inside the env block of the server entry, not at the top level of the config file, and that the key still exists under Settings > API Keys in your dashboard. A scrape “timed out.” The job is still running server-side and nothing is lost. The error includes the job ID, and the assistant will fetch the result with spidra_check_scrape_status. Bot-protected sites can take a couple of minutes. Results come back empty when using a schema. Check the schema: every field you want must be defined with a type. An object with no properties gives the AI nothing to fill in. npx cannot find the package. Make sure you are on Node 20 or newer and that your network allows access to the npm registry.

LangChain

Use Spidra as a document loader or agent tool inside LangChain pipelines.

SDKs Overview

Prefer code over chat? Browse the official SDKs.

Overview

Official SDKs

AI Frameworks

Before you start

Setup

Try it

How to choose a tool

The tools in detail

How it protects your credits

Output size

Configuration

Troubleshooting

LangChain

SDKs Overview

​Before you start

​Setup

​Try it

​How to choose a tool

​The tools in detail

​How it protects your credits

​Output size

​Configuration

​Troubleshooting

LangChain

SDKs Overview

Before you start

Setup

Try it

How to choose a tool

The tools in detail

How it protects your credits

Output size

Configuration

Troubleshooting