> ## Documentation Index > Fetch the complete documentation index at: https://docs.spidra.io/llms.txt > Use this file to discover all available pages before exploring further. # Get Crawl Report > See how a crawl understood its instruction, how many pages it read to find the ones it returned, and every link it did not follow, with the reason. Every finished crawl has a report. Use it when a page you expected is missing from the results: it lists every link the crawl didn't follow, why, and the page the link was on. ## Example Request ```bash cURL theme={null} curl "https://api.spidra.io/api/crawl/abc-123/report?reason=not_relevant&limit=50" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```typescript Node.js theme={null} const { report } = await spidra.crawlReport("abc-123", { reason: "not_relevant", limit: 50 }); ``` ```python Python theme={null} res = client.crawl_report("abc-123", reason="not_relevant", limit=50) ``` ## Response A real report for a crawl of three book pages from a category page: ```json theme={null} { "status": "success", "report": { "understoodAs": "Individual book detail pages under the mystery category.", "navigationPages": 1, "skippedTotal": 51, "skipped": [ { "reason": "not_relevant", "count": 51, "description": "Didn't match the crawl instruction" } ], "skippedUrls": [ { "url": "https://books.toscrape.com/index.html", "reason": "not_relevant", "foundOn": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" } ], "skippedUrlsTotal": 51 } } ``` | Field | Description | | - | - | | `understoodAs` | The kind of page Spidra looked for, in its own words. If this isn't what you meant, reword the crawl instruction. | | `navigationPages` | Pages read only to find the ones returned, such as category or listing pages. Not returned and not charged. | | `skippedTotal` | Links not followed, across all reasons. | | `skipped` | Each reason with its count and a plain-language description. | | `skippedUrls` | The skipped links, filtered by `reason` and paged by `limit` and `offset`. Up to 2,000 are recorded per crawl. | `report` is `null` while the crawl is still running, and for crawls that ran before reports existed. ## Reasons | Reason | Meaning | | - | - | | `not_relevant` | The link didn't lead to the kind of page the instruction asked for. | | `outside_domain` | Another site, or a subdomain while `allowSubdomains` is off. | | `include_pattern` | Didn't match `includePaths`. | | `exclude_pattern` | Matched `excludePaths`. | | `file_type` | A file rather than a page, such as an image, video, archive or script. | | `other_version_or_language` | The same page for another software version or locale, when the crawl started from a specific one. | | `duplicate_page` | Same content as a page already read, under another URL. | | `redirected_off_site` | Redirected outside the crawl's scope. | | `robots_txt` | Disallowed by the site's robots.txt, with `respectRobots: true`. | | `action_link` | A link that performs an action, such as add to cart or log out. These are never followed. | ## OpenAPI ````yaml GET /crawl/{jobId}/report openapi: 3.1.0 info: title: Spidra API version: 1.0.0 description: >- Public API endpoints for web scraping via Spidra. Authenticate with `Authorization: Bearer YOUR_API_KEY`. servers: - url: https://api.spidra.io/api security: - BearerAuth: [] - ApiKeyAuth: [] paths: /crawl/{jobId}/report: get: tags: - Crawling summary: Get Crawl Report description: >- Explains a finished crawl: how it understood the crawl instruction, how many pages it read only to find the wanted ones, and every link it did not follow, grouped by reason, with the page each link was found on. parameters: - name: jobId in: path required: true schema: type: string description: The crawl job ID. - name: reason in: query required: false schema: type: string enum: - not_relevant - outside_domain - include_pattern - exclude_pattern - file_type - other_version_or_language - duplicate_page - redirected_off_site - robots_txt - action_link description: Only list skipped URLs with this reason. - name: limit in: query required: false schema: type: integer default: 200 minimum: 1 maximum: 2000 description: How many skipped URLs to return. - name: offset in: query required: false schema: type: integer minimum: 0 default: 0 description: How many skipped URLs to skip, for paging. responses: '200': description: >- The crawl report. `report` is null while the crawl is running, and for crawls that ran before reports existed. content: application/json: schema: type: object properties: status: type: string enum: - success report: type: - object - 'null' properties: understoodAs: type: - string - 'null' description: The kind of page Spidra looked for, in its own words. navigationPages: type: integer description: >- Pages read only to find the wanted ones. Not returned and not charged. skippedTotal: type: integer description: Links not followed, across all reasons. skipped: type: array items: type: object properties: reason: type: string enum: - not_relevant - outside_domain - include_pattern - exclude_pattern - file_type - other_version_or_language - duplicate_page - redirected_off_site - robots_txt - action_link count: type: integer description: type: string description: The reason in plain language. skippedUrls: type: array items: type: object properties: url: type: string reason: type: string enum: - not_relevant - outside_domain - include_pattern - exclude_pattern - file_type - other_version_or_language - duplicate_page - redirected_off_site - robots_txt - action_link foundOn: type: - string - 'null' description: The page the link was found on. skippedUrlsTotal: type: integer description: >- Skipped URLs recorded for this filter, for paging. Up to 2,000 are recorded per crawl. message: type: string example: status: success report: understoodAs: Individual book detail pages under the mystery category. navigationPages: 1 skippedTotal: 51 skipped: - reason: not_relevant count: 51 description: Didn't match the crawl instruction skippedUrls: - url: https://books.toscrape.com/index.html reason: not_relevant foundOn: >- https://books.toscrape.com/catalogue/category/books/mystery_3/index.html skippedUrlsTotal: 51 '401': description: Invalid or missing API key content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: status: error message: Access token invalid or expired '404': description: Crawl job not found content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: status: error message: Crawl job not found components: schemas: ErrorResponse: type: object properties: status: type: string enum: - error message: type: string required: - status - message securitySchemes: BearerAuth: type: http scheme: bearer bearerFormat: API key ApiKeyAuth: type: apiKey in: header name: x-api-key ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.