> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# List Crawl History

> Retrieve a paginated list of your past crawl jobs, including status, page counts, and credit usage.

Use this endpoint to browse all the crawl jobs your account has submitted. Each record shows the base URL, how many pages were crawled, the current job status, and how many credits were consumed. This is useful for building dashboards, auditing your usage, or picking up a job ID you want to work with in a follow-up request.

## Pagination

The response is paginated. Use the `page` and `limit` query parameters to navigate through your history.

<CodeGroup>
  ```bash cURL theme={null}
  # First page (default: 10 results)
  curl https://api.spidra.io/api/crawl/history \
    -H "x-api-key: YOUR_API_KEY"

  # Second page with 25 results per page
  curl "https://api.spidra.io/api/crawl/history?page=2&limit=25" \
    -H "x-api-key: YOUR_API_KEY"
  ```

  ```python Python theme={null}
  import requests

  # First page
  response = requests.get(
      "https://api.spidra.io/api/crawl/history",
      headers={"x-api-key": "YOUR_API_KEY"}
  )

  # Second page with 25 results per page
  response = requests.get(
      "https://api.spidra.io/api/crawl/history",
      headers={"x-api-key": "YOUR_API_KEY"},
      params={"page": 2, "limit": 25}
  )
  ```

  ```javascript Node.js theme={null}
  // First page
  const response = await fetch("https://api.spidra.io/api/crawl/history", {
    headers: { "x-api-key": "YOUR_API_KEY" }
  });

  // Second page with 25 results per page
  const response = await fetch(
    "https://api.spidra.io/api/crawl/history?page=2&limit=25",
    { headers: { "x-api-key": "YOUR_API_KEY" } }
  );
  ```
</CodeGroup>

## Response Fields

| Field                  | Type    | Description                                                                   |
| ---------------------- | ------- | ----------------------------------------------------------------------------- |
| `jobs`                 | array   | List of crawl job records for this page                                       |
| `jobs[].id`            | string  | Unique job ID. Use this to call other endpoints like GET /crawl/{jobId}/pages |
| `jobs[].base_url`      | string  | The starting URL that was crawled                                             |
| `jobs[].status`        | string  | Job status: `waiting`, `active`, `completed`, or `failed`                     |
| `jobs[].max_pages`     | integer | The maximum number of pages requested                                         |
| `jobs[].pages_crawled` | integer | Actual number of pages successfully crawled                                   |
| `jobs[].created_at`    | string  | ISO 8601 timestamp when the job was created                                   |
| `jobs[].credits_used`  | number  | Credits charged to your account for this job                                  |
| `total`                | integer | Total number of crawl jobs in your account                                    |
| `page`                 | integer | The current page number                                                       |
| `totalPages`           | integer | Total number of pages available                                               |

## Example Response

```json theme={null}
{
  "jobs": [
    {
      "id": "abc-123",
      "base_url": "https://example.com/blog",
      "status": "completed",
      "max_pages": 10,
      "pages_crawled": 8,
      "created_at": "2025-12-17T15:00:00Z",
      "credits_used": 25
    },
    {
      "id": "def-456",
      "base_url": "https://store.example.com/products",
      "status": "failed",
      "max_pages": 20,
      "pages_crawled": 3,
      "created_at": "2025-12-16T09:30:00Z",
      "credits_used": 8
    }
  ],
  "total": 34,
  "page": 1,
  "totalPages": 4
}
```

<Tip>
  To get the full extracted data for a completed job, call [GET /crawl/{jobId}/pages](/api-reference/crawling/crawl-pages) using the `id` from this response.
</Tip>


## OpenAPI

````yaml GET /crawl/history
openapi: 3.1.0
info:
  title: Spidra API
  version: 1.0.0
  description: >-
    Public API endpoints for web scraping via Spidra. Authentication is via API
    key passed in the `x-api-key` header.
servers:
  - url: https://api.spidra.io/api
security:
  - ApiKeyAuth: []
paths:
  /crawl/history:
    get:
      tags:
        - Crawling
      summary: List Crawl History
      parameters:
        - name: page
          in: query
          schema:
            type: integer
            default: 1
            minimum: 1
        - name: limit
          in: query
          schema:
            type: integer
            default: 10
            minimum: 1
            maximum: 100
      responses:
        '200':
          description: Paginated list of crawl jobs
          content:
            application/json:
              schema:
                type: object
                properties:
                  jobs:
                    type: array
                    items:
                      type: object
                      properties:
                        id:
                          type: string
                        base_url:
                          type: string
                        status:
                          type: string
                        max_pages:
                          type: integer
                        pages_crawled:
                          type: integer
                        created_at:
                          type: string
                          format: date-time
                        credits_used:
                          type: number
                  total:
                    type: integer
                  page:
                    type: integer
                  totalPages:
                    type: integer
              example:
                jobs:
                  - id: abc-123
                    base_url: https://example.com
                    status: completed
                    max_pages: 10
                    pages_crawled: 8
                    created_at: '2025-12-17T15:00:00Z'
                    credits_used: 25
                total: 15
                page: 1
                totalPages: 2
        '401':
          description: Invalid or missing API key
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              example:
                status: error
                message: Access token invalid or expired
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        status:
          type: string
          enum:
            - error
        message:
          type: string
      required:
        - status
        - message
  securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: x-api-key

````