Skip to main content
POST
Map a Site
Map lists a site’s pages from its sitemap and the links on its pages. It doesn’t scrape anything, so it answers in a few seconds. Use it to see what a site contains, to find the exact pages you want before a batch scrape, or to pick the right starting point for a crawl. Unlike crawl and batch scrape, there’s no job to poll. The URLs come back in the response.

Example Request

Response

A real response, shortened to the first three links:

How Map finds URLs

  1. The sitemap. Spidra reads the Sitemap lines in the site’s robots.txt, then /sitemap.xml and /sitemap_index.xml, follows sitemap indexes and reads gzipped sitemaps.
  2. The start page. Its links, with their text.
  3. More pages. Spidra reads up to 40 more pages for links the start page doesn’t show (12 when the sitemap already lists 1,000 URLs or more). With a search, it reads pages about your search first, because they usually link to the rest of that topic. Without one, it reads index, contents and section pages first.
URLs are kept within the site (subdomains only with includeSubdomains), duplicates and links that perform actions such as log out are dropped, and current pages are listed before other versions and translations of them.

Tips

  • Use sitemap: "only" for speed on sites with a complete sitemap. It skips reading pages.
  • Use sitemap: "skip" when a sitemap is out of date.
  • Search in the site’s language. search matches the words in URLs and link text, so Bezirk finds pages on a German site that district wouldn’t.
  • No URLs found? The site may block automated visits or build its links with scripts. A crawl with Stealth Mode reads pages in a real browser.

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json
url
string<uri>
required

The website to map.

Only return URLs related to these words, best matches first. Matches the URL and the link text, in any language.

Maximum string length: 500
limit
integer
default:500

Most URLs returned.

Required range: 1 <= x <= 5000
sitemap
enum<string>
default:include

'include' reads the sitemap and the links on the site's pages, 'only' reads the sitemap only (fastest), 'skip' reads links only.

Available options:
include,
only,
skip
includeSubdomains
boolean
default:false

Include URLs on subdomains of the site.

ignoreQueryParams
boolean
default:true

Treat URLs that differ only by query string as one.

respectRobots
boolean
default:false

Leave out URLs the site's robots.txt disallows for crawlers.

cookies
string

Session cookies, for pages behind a login.

Response

The URLs found. When nothing is found, links is empty, creditsUsed is 0 and message says why.

status
enum<string>
Available options:
success
total
integer

URLs that matched the search (all URLs found when there is no search).

scanned
integer

All URLs found on the site, before the search filter.

sources
object

How many URLs came from the sitemap and from page links.

creditsUsed
integer
durationMs
integer
message
string