> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Map a Site

> List the page URLs of a website without scraping them, optionally ranked by a search. Answers in the same request for 1 credit.

Map lists a site's pages from its sitemap and the links on its pages. It doesn't scrape anything, so it answers in a few seconds. Use it to see what a site contains, to find the exact pages you want before a [batch scrape](/api-reference/scraping/batch-scrape), or to pick the right starting point for a [crawl](/api-reference/crawling/crawl).

Unlike crawl and batch scrape, there's no job to poll. The URLs come back in the response.

## Example Request

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.spidra.io/api/map \
    -H "Authorization: Bearer YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://docs.python.org/3/",
      "search": "asyncio",
      "limit": 20
    }'
  ```

  ```typescript Node.js theme={null}
  import { SpidraClient } from "spidra";

  const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY });

  const { links, total, scanned } = await spidra.map({
    url: "https://docs.python.org/3/",
    search: "asyncio",
    limit: 20,
  });
  ```

  ```python Python theme={null}
  from spidra import Spidra

  client = Spidra(api_key="YOUR_API_KEY")

  result = client.map("https://docs.python.org/3/", search="asyncio", limit=20)
  for link in result.links:
      print(link.url, link.title)
  ```
</CodeGroup>

## Response

A real response, shortened to the first three links:

```json theme={null}
{
  "status": "success",
  "links": [
    { "url": "https://docs.python.org/3/library/asyncio.html", "title": "asyncio" },
    { "url": "https://docs.python.org/3/library/asyncio-threading.html", "title": "asyncio and free-threaded Python" },
    { "url": "https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html", "title": "A conceptual overview of asyncio" }
  ],
  "total": 21,
  "scanned": 594,
  "sources": { "sitemap": 7, "pages": 587 },
  "creditsUsed": 1,
  "durationMs": 4914
}
```

| Field | Description |
| - | - |
| `links` | The URLs, best matches first when you pass `search`. `title` is the link text the URL was found with, when it came from a page. URLs from the sitemap only have no `title`. |
| `total` | How many URLs matched the search. Without a search, every URL found. |
| `scanned` | Every URL found on the site, before the search filter. |
| `sources` | How many URLs came from the sitemap and from page links. |
| `creditsUsed` | `1`, or `0` when nothing was found. |
| `message` | Present when no URLs were found, saying why. |

## How Map finds URLs

1. **The sitemap.** Spidra reads the `Sitemap` lines in the site's robots.txt, then `/sitemap.xml` and `/sitemap_index.xml`, follows sitemap indexes and reads gzipped sitemaps.
2. **The start page.** Its links, with their text.
3. **More pages.** Spidra reads up to 40 more pages for links the start page doesn't show (12 when the sitemap already lists 1,000 URLs or more). With a `search`, it reads pages about your search first, because they usually link to the rest of that topic. Without one, it reads index, contents and section pages first.

URLs are kept within the site (subdomains only with `includeSubdomains`), duplicates and links that perform actions such as log out are dropped, and current pages are listed before other versions and translations of them.

## Tips

* **Use `sitemap: "only"` for speed** on sites with a complete sitemap. It skips reading pages.
* **Use `sitemap: "skip"`** when a sitemap is out of date.
* **Search in the site's language.** `search` matches the words in URLs and link text, so `Bezirk` finds pages on a German site that `district` wouldn't.
* **No URLs found?** The site may block automated visits or build its links with scripts. A [crawl](/api-reference/crawling/crawl) with Stealth Mode reads pages in a real browser.


## OpenAPI

````yaml POST /map
openapi: 3.1.0
info:
  title: Spidra API
  version: 1.0.0
  description: >-
    Public API endpoints for web scraping via Spidra. Authenticate with
    `Authorization: Bearer YOUR_API_KEY`.
servers:
  - url: https://api.spidra.io/api
security:
  - BearerAuth: []
  - ApiKeyAuth: []
paths:
  /map:
    post:
      tags:
        - Crawling
      summary: Map a Site
      description: >-
        Lists the page URLs of a website without scraping them, from its sitemap
        and the links on its pages. Answers in the same request, usually in a
        few seconds. Costs 1 credit, and nothing when no URLs are found.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - url
              properties:
                url:
                  type: string
                  format: uri
                  description: The website to map.
                search:
                  type: string
                  maxLength: 500
                  description: >-
                    Only return URLs related to these words, best matches first.
                    Matches the URL and the link text, in any language.
                limit:
                  type: integer
                  default: 500
                  minimum: 1
                  maximum: 5000
                  description: Most URLs returned.
                sitemap:
                  type: string
                  enum:
                    - include
                    - only
                    - skip
                  default: include
                  description: >-
                    'include' reads the sitemap and the links on the site's
                    pages, 'only' reads the sitemap only (fastest), 'skip' reads
                    links only.
                includeSubdomains:
                  type: boolean
                  default: false
                  description: Include URLs on subdomains of the site.
                ignoreQueryParams:
                  type: boolean
                  default: true
                  description: Treat URLs that differ only by query string as one.
                respectRobots:
                  type: boolean
                  default: false
                  description: Leave out URLs the site's robots.txt disallows for crawlers.
                cookies:
                  type: string
                  description: Session cookies, for pages behind a login.
            example:
              url: https://docs.python.org/3/
              search: asyncio
              limit: 20
      responses:
        '200':
          description: >-
            The URLs found. When nothing is found, `links` is empty,
            `creditsUsed` is 0 and `message` says why.
          content:
            application/json:
              schema:
                type: object
                properties:
                  status:
                    type: string
                    enum:
                      - success
                  links:
                    type: array
                    items:
                      type: object
                      properties:
                        url:
                          type: string
                        title:
                          type: string
                          description: >-
                            The link text the URL was found with, when it came
                            from a page.
                  total:
                    type: integer
                    description: >-
                      URLs that matched the search (all URLs found when there is
                      no search).
                  scanned:
                    type: integer
                    description: All URLs found on the site, before the search filter.
                  sources:
                    type: object
                    properties:
                      sitemap:
                        type: integer
                      pages:
                        type: integer
                    description: How many URLs came from the sitemap and from page links.
                  creditsUsed:
                    type: integer
                  durationMs:
                    type: integer
                  message:
                    type: string
              example:
                status: success
                links:
                  - url: https://docs.python.org/3/library/asyncio.html
                    title: asyncio
                  - url: https://docs.python.org/3/library/asyncio-threading.html
                    title: asyncio and free-threaded Python
                  - url: >-
                      https://docs.python.org/3/howto/a-conceptual-overview-of-asyncio.html
                    title: A conceptual overview of asyncio
                total: 21
                scanned: 594
                sources:
                  sitemap: 7
                  pages: 587
                creditsUsed: 1
                durationMs: 4914
        '401':
          description: Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                status: error
                message: Access token invalid or expired
        '403':
          description: Monthly credit limit reached
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                status: error
                message: You have exceeded your monthly credit limit.
        '422':
          description: Invalid request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '429':
          description: More than 60 map requests in a minute
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                status: error
                message: Too many map requests. Please slow down and try again shortly.
        '502':
          description: The site could not be mapped. Not charged.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                status: error
                message: >-
                  The site took too long to map. Try again, or use sitemap:
                  "only".
                creditsUsed: 0
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        status:
          type: string
          enum:
            - error
        message:
          type: string
      required:
        - status
        - message
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      bearerFormat: API key
    ApiKeyAuth:
      type: apiKey
      in: header
      name: x-api-key

````

This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.