Before you start
You need:- A Spidra account. Sign up at app.spidra.io.
- An MCP-compatible client. Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Antigravity, OpenCode, and other MCP clients can connect to Spidra.
spd_.
Connect a client
The hosted server needs no install: point your client athttps://mcp.spidra.io/mcp and sign in with OAuth, or send an API key as a bearer token if your client doesn’t support OAuth. See Connect your client for the exact steps for Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Antigravity, Codex, GitHub Copilot CLI, and OpenCode.
Run the server locally
You can also run the Spidra MCP server on your own machine over stdio: useful if your client doesn’t support remote MCP servers, you want to manage the process yourself, or you need to point it at a staging or custom Spidra API throughSPIDRA_API_URL.
Local setups require Node.js 20 or newer.
Each configuration below starts npx -y spidra-mcp and passes your Spidra API key to the process through SPIDRA_API_KEY.
- Claude Code
- Claude Desktop
- Cursor
- VS Code
- Windsurf
- Antigravity
- OpenCode
Run:Start a new Claude Code session and run
/mcp to confirm that Spidra is connected.Try it
Once Spidra is connected, ask for web data in normal language.“Search for the best espresso machines of 2026 and summarize the top 3.”
“Scrape https://news.ycombinator.com and give me the top five stories with their points.”
“Compare the pricing pages for stripe.com and paddle.com and tell me which is cheaper for a small SaaS.”
“Here are 12 product URLs. Get the name, price, and rating from each one and return a table.”
“Crawl the first 10 pages of docs.spidra.io and summarize what the product does.”If the assistant returns data from the requested pages, the connection is working. The assistant handles tool selection, polling, and result retrieval automatically.
Choose the right tool
The first question is whether you already know the URL.- Use
spidra_searchwhen you don’t know the URL yet. It finds pages, news, images, videos, academic papers, or GitHub issues and PRs for a query, and can optionally fetch each web result’s actual page content in the same call. - Use
spidra_scrapewhen you know one page’s URL. It takes a single URL. - Use
spidra_batch_scrapefor anything involving more than one URL, even just two, even to compare them, each URL is processed independently and gets its own result. It accepts 2 to 50 URLs. - Use
spidra_parsefor documents rather than web pages: a PDF, Word, Excel or PowerPoint file, from a link or a small file the user shared. It returns Markdown page by page, and JSON if you ask for fields. - Use
spidra_mapto see which pages a site has. It lists a site’s URLs in a few seconds for 1 credit, ranked by a search if you give one, so you can scrape just the right pages. - Use
spidra_crawlwhen you know where to start but do not already know every page URL. Spidra discovers pages by following links according to your instructions.
Credit-efficient behavior
MCP agents can repeat actions if the server does not give them enough context about what is already running. Spidra’s MCP tools are designed to reduce unnecessary work and duplicate credit usage.- Tool descriptions guide the model toward the most appropriate operation for each request.
- Crawl guidance encourages smaller
maxPagesvalues when a large crawl is not necessary. - Long-running operations return job IDs and explicit polling instructions instead of encouraging the model to submit the same job again.
- Timeout responses tell the assistant that the existing job is still running and should be checked, not restarted.
- Rate-limit errors tell the assistant how long to wait before retrying.
- Validation errors identify what needs to change.
- Permanent errors tell the assistant when repeating the same request will not help.
- Cancelling a batch or crawl refunds credits for work that has not yet been processed.
- Retrying a batch or crawl page re-runs only the failed items, not the whole job.
Output limits
Large pages and crawls can return more content than an AI model can use effectively in one tool response. To keep responses manageable:- individual strings longer than 5,000 characters are truncated
- a single MCP tool response is capped at 80,000 characters
- truncated values are marked so the assistant knows content was omitted
spidra_crawl_pages. It returns signed download links to the full HTML and Markdown files.
Configuration
These environment variables apply when you run the MCP server locally or host your own instance. They are not required when you use the hosted endpoint with OAuth.
For HTTP connections, an API key can also be supplied per request:
Troubleshooting
The assistant does not see any Spidra tools. Check that the MCP server appears as connected in your client’s MCP settings. If you edited a configuration file manually, reload the MCP server or restart the client. In Claude Code, run/mcp to inspect the connection.
“No Spidra API key configured.” For a local stdio setup, make sure SPIDRA_API_KEY is inside the server’s environment configuration, for example:
Authorization: Bearer spd_YOUR_API_KEY. Also confirm that the key still exists under API Keys in your Spidra dashboard.
The hosted server returns a 401. The API key is missing, malformed, or no longer valid. Check that the header name is exactly Authorization, the value starts with Bearer, the key starts with spd_, and the key has not been revoked. If necessary, create a new key from the Spidra dashboard.
OAuth does not start. Confirm that your client supports OAuth for remote MCP servers. If the client does not support OAuth, use an API key or run the MCP server locally. If authentication started but failed, begin a fresh connection rather than reusing an old authorization URL.
OAuth reports an invalid or expired transaction. The authorization request may have expired or been interrupted; OAuth transactions expire after 10 minutes. Remove and reconnect the Spidra server, then complete the new authentication flow promptly. If the problem continues, use the API-key setup for that client and contact Spidra support with the error details.
A scrape “timed out.” This does not mean the job failed. The job continues running on Spidra, and the timeout response includes its job ID so the assistant can retrieve it with spidra_check_scrape_status. A timed-out wait window is not a reason to submit the same scrape again. Bot-protected sites can take longer to process.
Structured results are empty. Check the schema you provided. Every field you expect Spidra to return must be defined with a type:
npx cannot start the server. This only affects local setups. Confirm that node --version reports Node.js 20 or newer, and that your network can reach the npm registry. The hosted MCP server does not use npx and is unaffected by local Node.js issues.
