Map

Map lists a website’s pages without opening each one, and answers in the same request. Use it to see what a site contains, to find the exact pages you want before a batch scrape, or to pick a good starting point for a crawl.

Quick example

Find the pages about asyncio in the Python docs:

curl -X POST https://api.spidra.io/api/map \
  -H "Authorization: Bearer $SPIDRA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://docs.python.org/3/", "search": "asyncio", "limit": 20 }'

const { links, total, scanned } = await spidra.map({
  url: "https://docs.python.org/3/",
  search: "asyncio",
  limit: 20,
});

result = client.map("https://docs.python.org/3/", search="asyncio", limit=20)

for link in result.links:
    print(link.url, link.title)

The response, shortened to three links:

{
  "status": "success",
  "links": [
    { "url": "https://docs.python.org/3/library/asyncio.html", "title": "asyncio" },
    { "url": "https://docs.python.org/3/library/asyncio-threading.html", "title": "asyncio and free-threaded Python" },
    { "url": "https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html", "title": "A conceptual overview of asyncio" }
  ],
  "total": 21,
  "scanned": 594,
  "sources": { "sitemap": 7, "pages": 587 },
  "creditsUsed": 1,
  "durationMs": 4914
}

Spidra found 594 URLs on the site and returned the 21 about asyncio, best matches first.

What you get back

links: the URLs, best matches first when you search. title is the link text the URL was found with; URLs found only in the sitemap have no title.
total: how many URLs matched the search (every URL found, without a search).
scanned: every URL found before the search filter. It moves a little between runs.
sources: how many came from the sitemap and how many from page links.
message: present when nothing was found, saying why.

Options

Option	Default	What it does
`url` (required)		Any page on the site. The home page or a docs index gives the most complete list.
`search`		Only URLs related to these words, best matches first. Matches URLs and link text, in any language.
`limit`	500	1 to 5,000.
`sitemap`	`"include"`	`"include"` reads the sitemap and page links; `"only"` just the sitemap (fastest); `"skip"` just page links (for out-of-date sitemaps).
`includeSubdomains`	`false`	Also list subdomains, such as `docs.example.com` from `example.com`.
`ignoreQueryParams`	`true`	URLs that differ only by query string count as one.
`respectRobots`	`false`	Leave out URLs robots.txt disallows.
`cookies`		For pages behind a login.

How Map finds URLs

The sitemap: the Sitemap lines in robots.txt, then /sitemap.xml and /sitemap_index.xml, following sitemap indexes and gzipped sitemaps.
The start page’s links, with their text.
More pages: up to 40 more pages for links the start page doesn’t show (12 when the sitemap already lists 1,000 or more URLs). With a search, pages about your search are read first; without one, index and section pages first.

URLs stay on the site (subdomains only with includeSubdomains), duplicates and action links such as “log out” are dropped, and current pages come before other versions and translations of them. Map finds what’s in the sitemap or linked from pages. A page with neither won’t appear.

Cost

1 credit per map, however many URLs it returns. A map that finds nothing is free:

{
  "status": "success",
  "links": [],
  "total": 0,
  "creditsUsed": 0,
  "message": "This site doesn't publish a sitemap we could read. Try sitemap: \"include\" to read its links instead."
}

Tips

sitemap: "only" is fastest on sites with a complete sitemap.
Search in the site’s language. search matches the words in URLs and link text, so Bezirk finds pages on a German site that district wouldn’t.
No URLs? The site may block automated visits or build its links with scripts. A crawl with Stealth Mode reads pages in a real browser.
Map, then batch. Filter the links in your code (for example, URLs containing /docs/) and send them to Batch scrape. See Map a site and scrape only the pages you need.

In the dashboard

Open Map in the sidebar: Website, Look for (the search), Where to look, Max URLs, Include subdomains and Respect robots.txt. The results can be filtered, copied, exported as CSV or TXT, sent to a crawl, or scraped one by one. Past maps are under the form and in Logs.

The Map form on the left and 21 of 594 pages matching asyncio on docs.python.org on the right

Batch scrape

Scrape the URLs you found.

Crawl

Extract data from a section of the site.

Parse

Map a site’s PDFs, then parse them.

API reference

Every field and response.

Get started

Tools

Ready-made data

Getting the data you want

Supported sites

Guides

Dashboard

Account and billing

Quick example

What you get back

Options

How Map finds URLs

Cost

Tips

In the dashboard

Next

Batch scrape

Crawl

Parse

API reference

​Quick example

​What you get back

​Options

​How Map finds URLs

​Cost

​Tips

​In the dashboard

​Next

Batch scrape

Crawl

Parse

API reference

Quick example

What you get back

Options

How Map finds URLs

Cost

Tips

In the dashboard

Next