> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a RAG knowledge base from a docs site

> Crawl a documentation site into Markdown, drop the menus, and split it into chunks with their source URL, ready to embed in a vector database.

To answer questions about a product's docs with an LLM, you need the docs as clean text chunks, each with the page it came from. This guide [crawls](https://docs.spidra.io/crawl.md) a docs section as Markdown (no AI, so 1 credit per page), removes the navigation, and writes chunks to a JSON lines file you can embed.

You need `pip install spidra` and your API key in `SPIDRA_API_KEY`. The example uses the Python tutorial.

## 1. Crawl the section as Markdown

Start at the section you want; the crawl stays under that path. With no `transform_instruction` or schema, each page's `data` is its Markdown and no AI runs:

```python theme={null}
job = client.crawl(
    "https://docs.python.org/3/tutorial/",
    crawl_instruction="Every page of the tutorial",
    max_pages=10,
)
```

## 2. Split pages into chunks, without the menus

A crawl returns the whole page, including the site's navigation. Splitting at headings and skipping sections that are mostly short bullet links removes the menus. Here's the whole script:

```python theme={null}
# docs_to_chunks.py
import json
import os
import re

from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

# 1. Crawl the docs section as Markdown (no extraction, so no AI)
job = client.crawl(
    "https://docs.python.org/3/tutorial/",
    crawl_instruction="Every page of the tutorial",
    max_pages=10,
)
print(f"{len(job.result)} pages")

# 2. Split each page into chunks at its headings, skipping menus
def is_menu(section):
    lines = [l.strip() for l in section.splitlines() if l.strip()]
    short_bullets = sum(1 for l in lines if l.startswith(("* ", "- ")) and len(l) < 80)
    return not lines or short_bullets / len(lines) > 0.5

def chunks(markdown, max_chars=2000):
    sections = re.split(r"\n(?=#{1,3} )", markdown)
    for section in sections:
        if is_menu(section):
            continue
        for start in range(0, len(section), max_chars):
            piece = section[start:start + max_chars].strip()
            if len(piece) > 200:
                yield piece

# 3. Save one JSON line per chunk, with its source
count = 0
with open("chunks.jsonl", "w") as out:
    for page in job.result:
        for i, text in enumerate(chunks(page.data)):
            out.write(json.dumps({"url": page.url, "title": page.title, "chunk": i, "text": text}) + "\n")
            count += 1
print(f"{count} chunks")
```

```text theme={null}
10 pages
167 chunks
```

The first line of `chunks.jsonl` (text shortened):

```json theme={null}
{"url": "https://docs.python.org/3/tutorial/appetite.html", "title": "1. Whetting Your Appetite — Python 3.15.1 documentation", "chunk": 0, "text": "# 1\\. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there’s some task you’d like to automate. For example, you may wish to perform a search-and-replace over a large number of text files..."}
```

Ten pages cost 10 credits.

## 3. Embed and store

Each line has the text and its source, so you can embed `text` with any embedding model and store `url` and `title` as metadata, which lets your app cite the page an answer came from. Re-run the script on a schedule to keep the knowledge base current, or [watch the pages](https://docs.spidra.io/monitors.md) with `alertOn: "page"` to know when they change.

## Bigger sites

* **Raise `max_pages`** up to your plan's limit (100 on Free, up to 10,000 on Pro). The Python SDK's `crawl()` reads every page, even past the 100 that `GET /crawl/{jobId}` holds.
* **Fence the crawl** with `include_paths=["/3/tutorial/*"]` or `exclude_paths=["/3/whatsnew/*"]` when the section links elsewhere.
* **Pick pages first** with [Map](https://docs.spidra.io/map.md), then [batch scrape](https://docs.spidra.io/batch-scrape.md) them with `extract_content_only=True`, which removes menus, headers and footers for you.

## Good to know

* The splitter here is deliberately simple: a `#` comment at the start of a line inside a code block is also treated as a heading. For production, use a Markdown-aware splitter.
* Pages the crawl read only to find others aren't returned or charged. If a page you expected is missing, read the [crawl report](https://docs.spidra.io/api-reference/crawling/crawl-report.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.