Map a site and scrape only the pages you need

site = client.map("https://docs.python.org/3/", search="asyncio", limit=50)
print(f"{site.total} of {site.scanned} URLs match")

21 of 595 URLs match

urls = [link.url for link in site.links if "/library/asyncio" in link.url][:10]

# map_then_scrape.py
import os

from spidra import Spidra

client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])

# 1. List the site's pages about asyncio
site = client.map("https://docs.python.org/3/", search="asyncio", limit=50)
print(f"{site.total} of {site.scanned} URLs match")

# 2. Keep only the library reference pages
urls = [link.url for link in site.links if "/library/asyncio" in link.url][:10]
print(f"Scraping {len(urls)} pages")

# 3. Scrape just those pages
batch = client.batch_scrape(
    urls,
    prompt="What does this page cover? Name the main classes or functions it documents.",
    schema={
        "type": "object",
        "properties": {
            "topic": {"type": "string", "description": "The page's subject in a few words"},
            "main_apis": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Names of the classes or functions the page documents",
            },
        },
    },
)

for item in batch.items:
    if item.status == "completed":
        print(f"- {item.result['topic']}: {', '.join(item.result['main_apis'][:4])}")

21 of 595 URLs match
Scraping 10 pages
- asyncio module overview and high-level/low-level API entry points: asyncio.run, asyncio.sleep
- asyncio and free-threaded Python: asyncio.run_coroutine_threadsafe, loop.call_soon_threadsafe, asyncio.Lock, asyncio.Event
- Developing with asyncio: asyncio.run(), loop.set_debug(), loop.call_soon(), loop.call_at()
- Coroutines and tasks in Python asyncio: asyncio.create_task, asyncio.TaskGroup, asyncio.sleep, asyncio.gather
- asyncio synchronization primitives: asyncio.Lock, asyncio.Event, asyncio.Condition, asyncio.Semaphore
- asyncio queues for asynchronous I/O: asyncio.Queue, asyncio.PriorityQueue, asyncio.LifoQueue, asyncio.QueueEmpty
- asyncio call graph introspection: asyncio.print_call_graph, asyncio.format_call_graph, asyncio.capture_call_graph, asyncio.future_add_to_awaited_by
- Command-line introspection tools for asyncio: asyncio.tools
- asyncio runners and execution primitives: asyncio.run, asyncio.Runner
- asyncio streams: asyncio.open_connection, asyncio.start_server, asyncio.open_unix_connection, asyncio.start_unix_server