> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Map

> List a website's page URLs in seconds without scraping them, optionally ranked by a search term. Answers in the same request for 1 credit.

Map lists a website's pages without opening each one, and answers in the same request. Use it to see what a site contains, to find the exact pages you want before a [batch scrape](https://docs.spidra.io/batch-scrape.md), or to pick a good starting point for a [crawl](https://docs.spidra.io/crawl.md).

## Quick example

Find the pages about `asyncio` in the Python docs:

```bash curl theme={null}
curl -X POST https://api.spidra.io/api/map \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://docs.python.org/3/", "search": "asyncio", "limit": 20 }'
```

```typescript Node.js theme={null}
const { links, total, scanned } = await spidra.map({
  url: "https://docs.python.org/3/",
  search: "asyncio",
  limit: 20,
});
```

```python Python theme={null}
result = client.map("https://docs.python.org/3/", search="asyncio", limit=20)

for link in result.links:
    print(link.url, link.title)
```

The response, shortened to three links:

```json theme={null}
{
  "status": "success",
  "links": [
    { "url": "https://docs.python.org/3/library/asyncio.html", "title": "asyncio" },
    { "url": "https://docs.python.org/3/library/asyncio-threading.html", "title": "asyncio and free-threaded Python" },
    { "url": "https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html", "title": "A conceptual overview of asyncio" }
  ],
  "total": 21,
  "scanned": 594,
  "sources": { "sitemap": 7, "pages": 587 },
  "creditsUsed": 1,
  "durationMs": 4914
}
```

Spidra found 594 URLs on the site and returned the 21 about asyncio, best matches first.

## What you get back

* **`links`**: the URLs, best matches first when you search. `title` is the link text the URL was found with; URLs found only in the sitemap have no title.
* **`total`**: how many URLs matched the search (every URL found, without a search).
* **`scanned`**: every URL found before the search filter. It moves a little between runs.
* **`sources`**: how many came from the sitemap and how many from page links.
* **`message`**: present when nothing was found, saying why.

## Options

| Option | Default | What it does |
| - | - | - |
| `url` (required) | | Any page on the site. The home page or a docs index gives the most complete list. |
| `search` | | Only URLs related to these words, best matches first. Matches URLs and link text, in any language. |
| `limit` | 500 | 1 to 5,000. |
| `sitemap` | `"include"` | `"include"` reads the sitemap and page links; `"only"` just the sitemap (fastest); `"skip"` just page links (for out-of-date sitemaps). |
| `includeSubdomains` | `false` | Also list subdomains, such as `docs.example.com` from `example.com`. |
| `ignoreQueryParams` | `true` | URLs that differ only by query string count as one. |
| `respectRobots` | `false` | Leave out URLs robots.txt disallows. |
| `cookies` | | For pages behind a login. |

## How Map finds URLs

1. **The sitemap**: the `Sitemap` lines in robots.txt, then `/sitemap.xml` and `/sitemap_index.xml`, following sitemap indexes and gzipped sitemaps.
2. **The start page's links**, with their text.
3. **More pages**: up to 40 more pages for links the start page doesn't show (12 when the sitemap already lists 1,000 or more URLs). With a `search`, pages about your search are read first; without one, index and section pages first.

URLs stay on the site (subdomains only with `includeSubdomains`), duplicates and action links such as "log out" are dropped, and current pages come before other versions and translations of them.

Map finds what's in the sitemap or linked from pages. A page with neither won't appear.

## Cost

1 credit per map, however many URLs it returns. A map that finds nothing is free:

```json theme={null}
{
  "status": "success",
  "links": [],
  "total": 0,
  "creditsUsed": 0,
  "message": "This site doesn't publish a sitemap we could read. Try sitemap: \"include\" to read its links instead."
}
```

## Tips

* **`sitemap: "only"`** is fastest on sites with a complete sitemap.
* **Search in the site's language.** `search` matches the words in URLs and link text, so `Bezirk` finds pages on a German site that `district` wouldn't.
* **No URLs?** The site may block automated visits or build its links with scripts. A [crawl](https://docs.spidra.io/crawl.md) with Stealth Mode reads pages in a real browser.
* **Map, then batch.** Filter the links in your code (for example, URLs containing `/docs/`) and send them to [Batch scrape](https://docs.spidra.io/batch-scrape.md). See [Map a site and scrape only the pages you need](https://docs.spidra.io/guides/map-then-scrape.md).

## In the dashboard

Open **Map** in the sidebar: **Website**, **Look for** (the search), **Where to look**, **Max URLs**, **Include subdomains** and **Respect robots.txt**. The results can be filtered, copied, exported as CSV or TXT, sent to a crawl, or scraped one by one. Past maps are under the form and in [Logs](https://docs.spidra.io/dashboard/logs.md).

![The Map form on the left and 21 of 594 pages matching asyncio on docs.python.org on the right](https://mintcdn.com/spidra/Ew-SZt3zJq_nwjCD/images/map-playground.webp?fit=max\&auto=format\&n=Ew-SZt3zJq_nwjCD\&q=85\&s=8d595f2d606cbc0627d7e843adb461df)

## Next

> **[Batch scrape](https://docs.spidra.io/batch-scrape.md)**
>
> Scrape the URLs you found.

> **[Crawl](https://docs.spidra.io/crawl.md)**
>
> Extract data from a section of the site.

> **[Parse](https://docs.spidra.io/parse.md)**
>
> Map a site's PDFs, then parse them.

> **[API reference](https://docs.spidra.io/api-reference/crawling/map.md)**
>
> Every field and response.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.