> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Get Crawl Job Status
> Poll a running crawl job for progress. Returns the full result set when the job completes.
Poll this endpoint after submitting a crawl job. While the job is running it returns progress information. Once the job completes, the `result` array contains one entry per successfully crawled page.
Poll every 2–5 seconds. Stop when `status` is `completed`, `failed`, or `cancelled`.
## Example Request
```bash cURL theme={null}
curl https://api.spidra.io/api/crawl/abc-123 \
-H "Authorization: Bearer YOUR_API_KEY"
```
```python Python theme={null}
import requests, time
while True:
r = requests.get(
"https://api.spidra.io/api/crawl/abc-123",
headers={"Authorization": "Bearer YOUR_API_KEY"}
).json()
if r["status"] in ("completed", "failed", "cancelled"):
break
time.sleep(3)
```
```javascript Node.js theme={null}
let data;
do {
await new Promise(r => setTimeout(r, 3000));
const res = await fetch("https://api.spidra.io/api/crawl/abc-123", {
headers: { Authorization: "Bearer YOUR_API_KEY" }
});
data = await res.json();
} while (!["completed", "failed", "cancelled"].includes(data.status));
```
## Response Fields
| Field | Type | Description |
| ----------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | string | `waiting`, `active`, `completed`, `failed`, or `cancelled`. |
| `progress` | object | Current progress message and page counts. Present while the job is running. |
| `progress.message` | string | A description of what the crawler is doing right now. |
| `progress.pagesCrawled` | integer | Number of pages processed so far. |
| `progress.maxPages` | integer | The page limit for this job. |
| `result` | array or null | The full page result set. Populated once `status` is `completed`. `null` while the job is still running. |
| `result[].url` | string | The URL of this page. |
| `result[].title` | string | Page title. |
| `result[].status` | string | `success` or `failed`. |
| `result[].data` | any | Extracted content. Contains AI-extracted data when a `transformInstruction` or `schema` was provided. Contains the raw page markdown when neither was set. |
| `result[].html` | string or null | Signed URL to the raw HTML snapshot (valid for 1 hour). |
| `result[].markdown` | string or null | Signed URL to the markdown version of this page (valid for 1 hour). |
| `error` | string or null | Error message if `status` is `failed`. |
## Example Responses
**While running:**
```json theme={null}
{
"status": "active",
"progress": {
"message": "Crawling page 3 of 10",
"pagesCrawled": 3,
"maxPages": 10
},
"result": null,
"error": null
}
```
**Completed:**
```json theme={null}
{
"status": "completed",
"progress": {
"message": "Crawl complete",
"pagesCrawled": 10,
"maxPages": 10
},
"result": [
{
"url": "https://example.com/blog/post-1",
"title": "First Post",
"status": "success",
"data": {
"title": "First Post",
"author": "John",
"date": "2025-01-01"
},
"html": "https://storage.spidra.io/signed/...",
"markdown": "https://storage.spidra.io/signed/..."
}
],
"error": null
}
```
Unlike the scrape endpoint, the `result` field here is an **array** — one object per page crawled. For richer metadata on each page (page IDs, error messages, per-page timestamps), use [GET /crawl/{jobId}/pages](/api-reference/crawling/crawl-pages) after the job completes.
## OpenAPI
````yaml GET /crawl/{jobId}
openapi: 3.1.0
info:
title: Spidra API
version: 1.0.0
description: >-
Public API endpoints for web scraping via Spidra. Authenticate with
`Authorization: Bearer YOUR_API_KEY`.
servers:
- url: https://api.spidra.io/api
security:
- BearerAuth: []
- ApiKeyAuth: []
paths:
/crawl/{jobId}:
get:
tags:
- Crawling
summary: Get Crawl Job Status
parameters:
- name: jobId
in: path
required: true
schema:
type: string
description: The job ID returned from POST /crawl
responses:
'200':
description: Job status and results
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum:
- waiting
- active
- completed
- failed
- cancelled
progress:
type: object
properties:
message:
type: string
pagesCrawled:
type: integer
maxPages:
type: integer
result:
type: array
items:
type: object
properties:
url:
type: string
title:
type: string
status:
type: string
data:
description: >-
Extracted content for this page. Contains
AI-extracted data when a transformInstruction or
schema was provided. Contains the raw page markdown
when neither was set.
html:
type: string
description: >-
Signed URL to the raw HTML snapshot (valid for 1
hour).
nullable: true
markdown:
type: string
description: >-
Signed URL to the markdown version of the page
(valid for 1 hour).
nullable: true
nullable: true
error:
type: string
nullable: true
example:
status: completed
progress:
message: Crawl complete
pagesCrawled: 5
maxPages: 10
result:
- url: https://example.com/blog/post-1
title: First Post
status: success
data:
title: First Post
author: John
date: '2025-01-01'
html: >-
https://storage.spidra.io/signed/crawl/abc-123/page1.html?...
markdown: >-
https://storage.spidra.io/signed/crawl/abc-123/page1.md?...
error: null
'403':
description: Not authorized to access this job
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
status: error
message: You do not have permission to access this job
'404':
description: Job not found
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
status: error
message: Crawl job not found
components:
schemas:
ErrorResponse:
type: object
properties:
status:
type: string
enum:
- error
message:
type: string
required:
- status
- message
securitySchemes:
BearerAuth:
type: http
scheme: bearer
bearerFormat: API key
ApiKeyAuth:
type: apiKey
in: header
name: x-api-key
````