> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Map a site and scrape only the pages you need

> List a site's URLs with Map, filter them in your code, and batch scrape just the pages you want, instead of crawling everything.

Crawling a whole site to get ten pages wastes time and credits. Instead, list the site's URLs with [Map](https://docs.spidra.io/map.md) (1 credit), pick the ones you want in your code, and [batch scrape](https://docs.spidra.io/batch-scrape.md) only those. This guide gets a one-line summary and the main APIs of every asyncio reference page in the Python docs.

You need `pip install spidra` and your API key in `SPIDRA_API_KEY`.

## 1. Map the site

`search` ranks the URLs by how well they match, using the words in the URL and the link text:

```python theme={null}
site = client.map("https://docs.python.org/3/", search="asyncio", limit=50)
print(f"{site.total} of {site.scanned} URLs match")
```

```text theme={null}
21 of 595 URLs match
```

## 2. Pick the pages you want

You have the URLs as a list, so filter them however you like. Here, only the library reference pages:

```python theme={null}
urls = [link.url for link in site.links if "/library/asyncio" in link.url][:10]
```

Each link also has a `title` (its link text), which helps when URLs alone don't say enough.

## 3. Scrape just those pages

```python theme={null}
# map_then_scrape.py
import os

from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

# 1. List the site's pages about asyncio
site = client.map("https://docs.python.org/3/", search="asyncio", limit=50)
print(f"{site.total} of {site.scanned} URLs match")

# 2. Keep only the library reference pages
urls = [link.url for link in site.links if "/library/asyncio" in link.url][:10]
print(f"Scraping {len(urls)} pages")

# 3. Scrape just those pages
batch = client.batch_scrape(
    urls,
    prompt="What does this page cover? Name the main classes or functions it documents.",
    schema={
        "type": "object",
        "properties": {
            "topic": {"type": "string", "description": "The page's subject in a few words"},
            "main_apis": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Names of the classes or functions the page documents",
            },
        },
    },
)

for item in batch.items:
    if item.status == "completed":
        print(f"- {item.result['topic']}: {', '.join(item.result['main_apis'][:4])}")
```

```text theme={null}
21 of 595 URLs match
Scraping 10 pages
- asyncio module overview and high-level/low-level API entry points: asyncio.run, asyncio.sleep
- asyncio and free-threaded Python: asyncio.run_coroutine_threadsafe, loop.call_soon_threadsafe, asyncio.Lock, asyncio.Event
- Developing with asyncio: asyncio.run(), loop.set_debug(), loop.call_soon(), loop.call_at()
- Coroutines and tasks in Python asyncio: asyncio.create_task, asyncio.TaskGroup, asyncio.sleep, asyncio.gather
- asyncio synchronization primitives: asyncio.Lock, asyncio.Event, asyncio.Condition, asyncio.Semaphore
- asyncio queues for asynchronous I/O: asyncio.Queue, asyncio.PriorityQueue, asyncio.LifoQueue, asyncio.QueueEmpty
- asyncio call graph introspection: asyncio.print_call_graph, asyncio.format_call_graph, asyncio.capture_call_graph, asyncio.future_add_to_awaited_by
- Command-line introspection tools for asyncio: asyncio.tools
- asyncio runners and execution primitives: asyncio.run, asyncio.Runner
- asyncio streams: asyncio.open_connection, asyncio.start_server, asyncio.open_unix_connection, asyncio.start_unix_server
```

The `description` on each schema field matters: without it, one page put its whole summary in `topic` and left `main_apis` empty.

The whole run cost 28 credits: 1 for the map and 27 for the ten pages (1 each, plus AI).

## Map or crawl?

| Use | When |
| - | - |
| Map, then batch scrape | You can pick the pages by their URL or link text. You see the list before you pay for any page. |
| [Crawl](https://docs.spidra.io/crawl.md) | The pages you want are easier to describe than to pick ("individual product pages"), or they're only found by following links. |

## If something goes wrong

* **Map found nothing.** It costs nothing. Try `sitemap="skip"` (the sitemap may be out of date), or crawl the site instead; some sites build their links with scripts.
* **Too many URLs match.** Lower `limit`, make `search` more specific, or filter harder in step 2.
* **Over 50 pages.** Send them in batches of 50, as in [Scrape a list of URLs to CSV](https://docs.spidra.io/guides/urls-to-csv.md#3-more-than-50-urls).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.