> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# LangChain

> Use Spidra as a document loader or agent tool inside LangChain pipelines.

`langchain-spidra` integrates Spidra into LangChain in two ways:

* **Document loader.** `SpidraLoader` pulls web content as LangChain `Document` objects, ready to feed into a vector store or retrieval chain.
* **Agent tools.** `SpidraScrape`, `SpidraCrawl`, and `SpidraBatchScrape` are `BaseTool` classes you can bind to any LangChain agent so it can browse the web on its own.

## Installation

```bash theme={null}
pip install langchain-spidra
```

Requires Python 3.9 or higher. This package is for LangChain's Python framework; if you are building agents in JavaScript, see the [Node SDK's AI agent integration](/sdks/node#ai-agent-integration) instead.

<Note>
  Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings** > **API Keys**.
  Set it as an environment variable so you never hardcode it.

  ```bash theme={null}
  export SPIDRA_API_KEY="spd_your-api-key"
  ```
</Note>

## Document loader

`SpidraLoader` implements LangChain's `BaseLoader` interface and returns a list of `Document` objects, each with `page_content` and `metadata`.

### Scrape a page

The default mode. Returns raw markdown, which is ideal for RAG indexing where you want clean, unmodified content:

```python theme={null}
from langchain_spidra import SpidraLoader

loader = SpidraLoader(url="https://example.com")
docs = loader.load()

print(docs[0].page_content[:200])
print(docs[0].metadata)  # {"source": "https://example.com"}
```

Pass up to three URLs at once and get one document per URL:

```python theme={null}
loader = SpidraLoader(
    urls=["https://example.com/pricing", "https://example.com/features"],
)
docs = loader.load()
# len(docs) == 2
```

#### AI extraction

Add a `prompt` to run AI extraction. You get back a single document containing the structured result instead of raw markdown:

```python theme={null}
loader = SpidraLoader(
    url="https://example.com/pricing",
    params={
        "prompt": "Extract all pricing plans: name, price, and included features",
        "output": "json",
    },
)
docs = loader.load()
# docs[0].page_content == '{"plans": [...]}'
# docs[0].metadata["ai_extraction_failed"] == False
```

### Crawl a site

Set `mode="crawl"` to crawl an entire site. Returns one document per discovered page, which is exactly what you want for indexing a docs site or knowledge base:

```python theme={null}
loader = SpidraLoader(
    url="https://docs.example.com",
    mode="crawl",
    params={
        "crawl_instruction": "Follow all documentation pages",
        "max_pages": 50,
    },
)
docs = loader.load()

for doc in docs:
    print(doc.metadata["source"], doc.page_content[:100])
```

<Note>
  Omitting `transform_instruction` returns raw markdown per page with no AI processing and no token charges. Pass it only when you need AI extraction on each crawled page.
</Note>

With AI extraction per page:

```python theme={null}
loader = SpidraLoader(
    url="https://competitor.com/blog",
    mode="crawl",
    params={
        "crawl_instruction": "Follow links to blog posts only",
        "transform_instruction": "Extract the post title, author, and a one-sentence summary",
        "max_pages": 30,
    },
)
docs = loader.load()
```

### Batch scrape

Set `mode="batch"` to scrape many URLs in parallel. Returns one document per completed URL:

```python theme={null}
loader = SpidraLoader(
    urls=[
        "https://shop.example.com/product/1",
        "https://shop.example.com/product/2",
        "https://shop.example.com/product/3",
    ],
    mode="batch",
)
docs = loader.load()
```

### Use with a vector store

The most common use case: crawl a site and build a searchable knowledge base.

```python theme={null}
from langchain_spidra import SpidraLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

loader = SpidraLoader(
    url="https://docs.example.com",
    mode="crawl",
    params={"crawl_instruction": "Crawl all documentation pages", "max_pages": 50},
)
docs = loader.load()

vectorstore = FAISS.from_documents(docs, OpenAIEmbeddings())
retriever = vectorstore.as_retriever()
```

### Lazy loading

For large crawls, use `lazy_load()` to process documents as they arrive instead of waiting for everything to finish:

```python theme={null}
loader = SpidraLoader(url="https://docs.example.com", mode="crawl")

for doc in loader.lazy_load():
    # process one document at a time
    print(doc.metadata["source"])
```

### Loader parameters

| Parameter | Type       | Default    | Description                                                               |
| --------- | ---------- | ---------- | ------------------------------------------------------------------------- |
| `url`     | str        | none       | URL to scrape or crawl (required for `scrape` and `crawl` modes)          |
| `urls`    | list\[str] | none       | List of URLs (required for `batch` mode; optional multi-URL for `scrape`) |
| `api_key` | str        | env var    | Spidra API key. Falls back to `SPIDRA_API_KEY`                            |
| `api_url` | str        | none       | Override the API base URL (self-hosted instances)                         |
| `mode`    | str        | `"scrape"` | One of `"scrape"`, `"crawl"`, or `"batch"`                                |
| `params`  | dict       | `{}`       | Extra parameters forwarded to the underlying SDK method                   |

***

## Agent tools

Each Spidra capability is available as a standalone `BaseTool` you can bind to an agent.

### Available tools

| Tool                | What it does                                                    |
| ------------------- | --------------------------------------------------------------- |
| `SpidraScrape`      | Scrapes a single URL and returns its markdown content           |
| `SpidraCrawl`       | Crawls a site from a starting URL, returns all discovered pages |
| `SpidraBatchScrape` | Scrapes a list of URLs in parallel, returns one result per URL  |

### Standalone usage

```python theme={null}
from langchain_spidra import SpidraScrape, SpidraCrawl, SpidraBatchScrape

# Scrape a page
scrape = SpidraScrape()
result = scrape.invoke({"url": "https://example.com"})
print(result["content"])

# Crawl a site
crawl = SpidraCrawl()
pages = crawl.invoke({"url": "https://docs.example.com", "max_pages": 20})
for page in pages:
    print(page["url"], page["data"][:100])

# Batch scrape
batch = SpidraBatchScrape()
items = batch.invoke({"urls": ["https://a.com", "https://b.com"]})
for item in items:
    print(item["url"], item["status"], item["result"][:100])
```

**What each tool returns:**

| Tool                | Returns                                                                                 |
| ------------------- | --------------------------------------------------------------------------------------- |
| `SpidraScrape`      | A dict: `content` (markdown or JSON string), `ai_extraction_failed`, `extraction_empty` |
| `SpidraCrawl`       | A list of page dicts: `url`, `data`, `title`                                            |
| `SpidraBatchScrape` | A list of item dicts: `url`, `result`, `status` (`"completed"` or `"failed"`)           |

**Error handling:** the tools never raise. On failure they return the error as a plain string, so the agent can read what went wrong and adjust. When calling a tool yourself, check the type before indexing into the result:

```python theme={null}
result = scrape.invoke({"url": "https://example.com"})
if isinstance(result, str):
    print("Scrape failed:", result)
else:
    print(result["content"])
```

### Bind to an agent

```python theme={null}
from langchain_anthropic import ChatAnthropic
from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.prompts import ChatPromptTemplate
from langchain_spidra import SpidraScrape, SpidraCrawl

tools = [SpidraScrape(), SpidraCrawl()]

llm = ChatAnthropic(model="claude-opus-4-8")

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful research assistant with access to web scraping tools."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

result = executor.invoke({
    "input": "What are the main features listed on https://spidra.io?"
})
print(result["output"])
```

### Tool parameters

**`SpidraScrape`**

| Parameter              | Type | Description                                       |
| ---------------------- | ---- | ------------------------------------------------- |
| `url`                  | str  | The URL to scrape                                 |
| `prompt`               | str  | AI extraction instruction (enables AI mode)       |
| `output`               | str  | Output format: `"markdown"` (default) or `"json"` |
| `use_proxy`            | bool | Route through a residential proxy                 |
| `proxy_country`        | str  | Two-letter country code for the proxy             |
| `extract_content_only` | bool | Strip nav and ads, return main content only       |

**`SpidraCrawl`**

| Parameter               | Type       | Description                                      |
| ----------------------- | ---------- | ------------------------------------------------ |
| `url`                   | str        | Starting URL for the crawl                       |
| `crawl_instruction`     | str        | Which pages to follow, in plain language         |
| `transform_instruction` | str        | What to extract from each page (enables AI mode) |
| `max_pages`             | int        | Maximum number of pages to crawl                 |
| `max_depth`             | int        | Maximum link depth from the starting URL         |
| `include_paths`         | list\[str] | URL path patterns to include                     |
| `exclude_paths`         | list\[str] | URL path patterns to skip                        |
| `use_proxy`             | bool       | Route through a residential proxy                |
| `proxy_country`         | str        | Two-letter country code for the proxy            |

**`SpidraBatchScrape`**

| Parameter              | Type       | Description                                       |
| ---------------------- | ---------- | ------------------------------------------------- |
| `urls`                 | list\[str] | List of URLs to scrape                            |
| `prompt`               | str        | AI extraction instruction applied to all pages    |
| `output`               | str        | Output format: `"markdown"` (default) or `"json"` |
| `use_proxy`            | bool       | Route through a residential proxy                 |
| `proxy_country`        | str        | Two-letter country code for the proxy             |
| `extract_content_only` | bool       | Strip nav and ads, return main content only       |

## Source

The package is open-source at [github.com/spidra-io/spidra-langchain](https://github.com/spidra-io/spidra-langchain). Bug reports and contributions welcome.

<CardGroup cols={2}>
  <Card title="Python SDK" icon="python" href="/sdks/python">
    Use the Spidra Python SDK directly without LangChain.
  </Card>

  <Card title="Node.js SDK" icon="node-js" href="/sdks/node">
    Official Node.js SDK for JavaScript and TypeScript projects.
  </Card>
</CardGroup>