LangChain

langchain-spidra integrates Spidra into LangChain in two ways:

Document loader. SpidraLoader pulls web content as LangChain Document objects, ready to feed into a vector store or retrieval chain.
Agent tools. SpidraScrape, SpidraCrawl, and SpidraBatchScrape are BaseTool classes you can bind to any LangChain agent so it can browse the web on its own.

Installation

pip install langchain-spidra

Requires Python 3.9 or higher. This package is for LangChain’s Python framework; if you are building agents in JavaScript, see the Node SDK’s AI agent integration instead.

Get your API key from app.spidra.io under Settings > API Keys. Set it as an environment variable so you never hardcode it.

export SPIDRA_API_KEY="spd_your-api-key"

Document loader

SpidraLoader implements LangChain’s BaseLoader interface and returns a list of Document objects, each with page_content and metadata.

Scrape a page

The default mode. Returns raw markdown, which is ideal for RAG indexing where you want clean, unmodified content:

from langchain_spidra import SpidraLoader

loader = SpidraLoader(url="https://example.com")
docs = loader.load()

print(docs[0].page_content[:200])
print(docs[0].metadata)  # {"source": "https://example.com"}

Pass up to three URLs at once and get one document per URL:

loader = SpidraLoader(
    urls=["https://example.com/pricing", "https://example.com/features"],
)
docs = loader.load()
# len(docs) == 2

AI extraction

Add a prompt to run AI extraction. You get back a single document containing the structured result instead of raw markdown:

loader = SpidraLoader(
    url="https://example.com/pricing",
    params={
        "prompt": "Extract all pricing plans: name, price, and included features",
        "output": "json",
    },
)
docs = loader.load()
# docs[0].page_content == '{"plans": [...]}'
# docs[0].metadata["ai_extraction_failed"] == False

Crawl a site

Set mode="crawl" to crawl an entire site. Returns one document per discovered page, which is exactly what you want for indexing a docs site or knowledge base:

loader = SpidraLoader(
    url="https://docs.example.com",
    mode="crawl",
    params={
        "crawl_instruction": "Follow all documentation pages",
        "max_pages": 50,
    },
)
docs = loader.load()

for doc in docs:
    print(doc.metadata["source"], doc.page_content[:100])

Omitting transform_instruction returns raw markdown per page with no AI processing and no token charges. Pass it only when you need AI extraction on each crawled page.

With AI extraction per page:

loader = SpidraLoader(
    url="https://competitor.com/blog",
    mode="crawl",
    params={
        "crawl_instruction": "Follow links to blog posts only",
        "transform_instruction": "Extract the post title, author, and a one-sentence summary",
        "max_pages": 30,
    },
)
docs = loader.load()

Batch scrape

Set mode="batch" to scrape many URLs in parallel. Returns one document per completed URL:

loader = SpidraLoader(
    urls=[
        "https://shop.example.com/product/1",
        "https://shop.example.com/product/2",
        "https://shop.example.com/product/3",
    ],
    mode="batch",
)
docs = loader.load()

Use with a vector store

The most common use case: crawl a site and build a searchable knowledge base.

from langchain_spidra import SpidraLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

loader = SpidraLoader(
    url="https://docs.example.com",
    mode="crawl",
    params={"crawl_instruction": "Crawl all documentation pages", "max_pages": 50},
)
docs = loader.load()

vectorstore = FAISS.from_documents(docs, OpenAIEmbeddings())
retriever = vectorstore.as_retriever()

Lazy loading

For large crawls, use lazy_load() to process documents as they arrive instead of waiting for everything to finish:

loader = SpidraLoader(url="https://docs.example.com", mode="crawl")

for doc in loader.lazy_load():
    # process one document at a time
    print(doc.metadata["source"])

Loader parameters

Parameter	Type	Default	Description
`url`	str	none	URL to scrape or crawl (required for `scrape` and `crawl` modes)
`urls`	list[str]	none	List of URLs (required for `batch` mode; optional multi-URL for `scrape`)
`api_key`	str	env var	Spidra API key. Falls back to `SPIDRA_API_KEY`
`api_url`	str	none	Override the API base URL (self-hosted instances)
`mode`	str	`"scrape"`	One of `"scrape"`, `"crawl"`, or `"batch"`
`params`	dict	`{}`	Extra parameters forwarded to the underlying SDK method

Agent tools

Each Spidra capability is available as a standalone BaseTool you can bind to an agent.

Available tools

Tool	What it does
`SpidraScrape`	Scrapes a single URL and returns its markdown content
`SpidraCrawl`	Crawls a site from a starting URL, returns all discovered pages
`SpidraBatchScrape`	Scrapes a list of URLs in parallel, returns one result per URL

Standalone usage

from langchain_spidra import SpidraScrape, SpidraCrawl, SpidraBatchScrape

# Scrape a page
scrape = SpidraScrape()
result = scrape.invoke({"url": "https://example.com"})
print(result["content"])

# Crawl a site
crawl = SpidraCrawl()
pages = crawl.invoke({"url": "https://docs.example.com", "max_pages": 20})
for page in pages:
    print(page["url"], page["data"][:100])

# Batch scrape
batch = SpidraBatchScrape()
items = batch.invoke({"urls": ["https://a.com", "https://b.com"]})
for item in items:
    print(item["url"], item["status"], item["result"][:100])

What each tool returns:

Tool	Returns
`SpidraScrape`	A dict: `content` (markdown or JSON string), `ai_extraction_failed`, `extraction_empty`
`SpidraCrawl`	A list of page dicts: `url`, `data`, `title`
`SpidraBatchScrape`	A list of item dicts: `url`, `result`, `status` (`"completed"` or `"failed"`)

Error handling: the tools never raise. On failure they return the error as a plain string, so the agent can read what went wrong and adjust. When calling a tool yourself, check the type before indexing into the result:

result = scrape.invoke({"url": "https://example.com"})
if isinstance(result, str):
    print("Scrape failed:", result)
else:
    print(result["content"])

Bind to an agent

from langchain_anthropic import ChatAnthropic
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain_spidra import SpidraScrape, SpidraCrawl

tools = [SpidraScrape(), SpidraCrawl()]

llm = ChatAnthropic(model="claude-opus-4-8")

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful research assistant with access to web scraping tools."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

result = executor.invoke({
    "input": "What are the main features listed on https://spidra.io?"
})
print(result["output"])

Tool parameters

SpidraScrape

Parameter	Type	Description
`url`	str	The URL to scrape
`prompt`	str	AI extraction instruction (enables AI mode)
`output`	str	Output format: `"markdown"` (default) or `"json"`
`use_proxy`	bool	Route through a residential proxy
`proxy_country`	str	Two-letter country code for the proxy
`extract_content_only`	bool	Strip nav and ads, return main content only

SpidraCrawl

Parameter	Type	Description
`url`	str	Starting URL for the crawl
`crawl_instruction`	str	Which pages to follow, in plain language
`transform_instruction`	str	What to extract from each page (enables AI mode)
`max_pages`	int	Maximum number of pages to crawl
`max_depth`	int	Maximum link depth from the starting URL
`include_paths`	list[str]	URL path patterns to include
`exclude_paths`	list[str]	URL path patterns to skip
`use_proxy`	bool	Route through a residential proxy
`proxy_country`	str	Two-letter country code for the proxy

SpidraBatchScrape

Parameter	Type	Description
`urls`	list[str]	List of URLs to scrape
`prompt`	str	AI extraction instruction applied to all pages
`output`	str	Output format: `"markdown"` (default) or `"json"`
`use_proxy`	bool	Route through a residential proxy
`proxy_country`	str	Two-letter country code for the proxy
`extract_content_only`	bool	Strip nav and ads, return main content only

Source

The package is open-source at github.com/spidra-io/spidra-langchain. Bug reports and contributions welcome.

Python SDK

Use the Spidra Python SDK directly without LangChain.

Node.js SDK

Official Node.js SDK for JavaScript and TypeScript projects.

Overview

Official SDKs

AI Frameworks

Installation

Document loader

Scrape a page

AI extraction

Crawl a site

Batch scrape

Use with a vector store

Lazy loading

Loader parameters

Agent tools

Available tools

Standalone usage

Bind to an agent

Tool parameters

Source

Python SDK

Node.js SDK

​Installation

​Document loader

​Scrape a page

​AI extraction

​Crawl a site

​Batch scrape

​Use with a vector store

​Lazy loading

​Loader parameters

​Agent tools

​Available tools

​Standalone usage

​Bind to an agent

​Tool parameters

​Source

Python SDK

Node.js SDK

Installation

Document loader

Scrape a page

AI extraction

Crawl a site

Batch scrape

Use with a vector store

Lazy loading

Loader parameters

Agent tools

Available tools

Standalone usage

Bind to an agent

Tool parameters

Source