> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Get Batch Status
> Poll for batch progress and retrieve per-item results
## Polling Pattern
Poll this endpoint every **2–5 seconds** after submitting a batch. Once `status` is `completed`, `failed`, or `cancelled`, stop polling (the batch will not change further).
```javascript Node.js theme={null}
async function pollBatch(batchId) {
while (true) {
const res = await fetch(
`https://api.spidra.io/api/batch/scrape/${batchId}`,
{ headers: { "x-api-key": "YOUR_API_KEY" } }
);
const data = await res.json();
console.log(`${data.completedCount}/${data.totalUrls} complete, ${data.failedCount} failed`);
if (["completed", "failed", "cancelled"].includes(data.status)) {
return data;
}
await new Promise((r) => setTimeout(r, 3000));
}
}
```
```python Python theme={null}
import time, requests
def poll_batch(batch_id):
headers = {"x-api-key": "YOUR_API_KEY"}
while True:
data = requests.get(
f"https://api.spidra.io/api/batch/scrape/{batch_id}",
headers=headers,
).json()
print(f"{data['completedCount']}/{data['totalUrls']} complete, {data['failedCount']} failed")
if data["status"] in ("completed", "failed", "cancelled"):
return data
time.sleep(3)
```
```bash cURL theme={null}
curl https://api.spidra.io/api/batch/scrape/YOUR_BATCH_ID \
-H "x-api-key: YOUR_API_KEY"
```
***
## Batch Status Values
| Status | Meaning |
| ----------- | ---------------------------------------------------------------------------- |
| `pending` | Queued — no items have started yet |
| `running` | At least one item is being processed |
| `completed` | All items reached a terminal state. Check `failedCount` for partial failures |
| `failed` | The batch failed unexpectedly |
| `cancelled` | Cancelled via `DELETE /api/batch/scrape/{batchId}` |
`completed` does not guarantee every URL succeeded. A batch is marked `completed` when all items have a terminal status (`completed` or `failed`). Always inspect `failedCount` and individual item statuses to detect partial failures.
***
## Response
**Example — batch in progress:**
```json theme={null}
{
"status": "running",
"totalUrls": 5,
"completedCount": 3,
"failedCount": 0,
"items": [...],
"createdAt": "2024-01-15T10:00:00Z",
"finishedAt": null
}
```
**Example — batch completed:**
```json theme={null}
{
"status": "completed",
"totalUrls": 5,
"completedCount": 4,
"failedCount": 1,
"items": [...],
"createdAt": "2024-01-15T10:00:00Z",
"finishedAt": "2024-01-15T10:00:42Z"
}
```
### Top-Level Fields
| Field | Type | Description |
| ---------------- | ------------------ | -------------------------------------------------------------------------------------- |
| `status` | `string` | Batch-level status: `pending`, `running`, `completed`, `failed`, or `cancelled` |
| `totalUrls` | `number` | Total number of URLs in the batch |
| `completedCount` | `number` | Items that finished successfully |
| `failedCount` | `number` | Items that encountered an error |
| `items` | `array` | Per-item results (see below) |
| `createdAt` | `string` | ISO 8601 timestamp when the batch was submitted |
| `finishedAt` | `string` \| `null` | ISO 8601 timestamp when the batch reached a terminal state, or `null` if still running |
***
## Per-Item Fields
Each entry in `items` represents one URL:
```json theme={null}
{
"uuid": "a1b2c3d4-0000-0000-0000-000000000000",
"url": "https://example.com/product/42",
"jobId": "bull-worker-job-id",
"status": "completed",
"result": {
"name": "Widget Pro",
"price": 49.99,
"available": true
},
"error": null,
"creditsUsed": 3,
"startedAt": "2024-01-15T10:00:05Z",
"finishedAt": "2024-01-15T10:00:11Z",
"screenshotUrl": null
}
```
| Field | Type | Description |
| --------------- | ------------------ | --------------------------------------------------------------------------------------------- |
| `uuid` | `string` | Unique ID for this batch item |
| `url` | `string` | The URL that was processed |
| `jobId` | `string` \| `null` | Internal worker job ID. `null` if the item is still pending |
| `status` | `string` | Item-level status: `pending`, `running`, `completed`, `failed`, or `cancelled` |
| `result` | `any` \| `null` | Extracted content. Object if `output: "json"`, string if `"markdown"`. `null` until completed |
| `error` | `string` \| `null` | Error description if `status` is `failed`, otherwise `null` |
| `creditsUsed` | `number` | Credits consumed by this item. `0` for failed or cancelled items |
| `startedAt` | `string` \| `null` | ISO 8601 timestamp when the worker started this item |
| `finishedAt` | `string` \| `null` | ISO 8601 timestamp when this item completed or failed |
| `screenshotUrl` | `string` \| `null` | S3 URL for the screenshot, or `null` if screenshots were not requested |
***
## Handling Partial Failures
When `completedCount + failedCount === totalUrls` but some items failed, retry them without re-running the whole batch:
```javascript theme={null}
const data = await pollBatch(batchId);
if (data.failedCount > 0) {
const retry = await fetch(
`https://api.spidra.io/api/batch/scrape/${batchId}/retry`,
{ method: "POST", headers: { "x-api-key": "YOUR_API_KEY" } }
);
const { retriedCount } = await retry.json();
console.log(`Retrying ${retriedCount} failed items...`);
}
```
***
## Errors
| Code | Reason |
| ----- | -------------------------------------------------------------- |
| `401` | Missing `x-api-key` header |
| `403` | Invalid or expired API key |
| `404` | No batch found with this ID, or it belongs to a different user |
***
Re-queue only the failed items
Stop processing and refund credits
## OpenAPI
````yaml GET /batch/scrape/{batchId}
openapi: 3.1.0
info:
title: Spidra API
version: 1.0.0
description: >-
Public API endpoints for web scraping via Spidra. Authentication is via API
key passed in the `x-api-key` header.
servers:
- url: https://api.spidra.io/api
security:
- ApiKeyAuth: []
paths:
/batch/scrape/{batchId}:
get:
tags:
- Batch Scraping
summary: Get Batch Status
description: >-
Poll for batch progress and retrieve per-item results. Poll every 2–5
seconds until status is completed, failed, or cancelled.
parameters:
- name: batchId
in: path
required: true
schema:
type: string
format: uuid
description: The batch ID returned by POST /batch/scrape
responses:
'200':
description: Batch status and per-item results
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum:
- pending
- running
- completed
- failed
- cancelled
totalUrls:
type: integer
completedCount:
type: integer
failedCount:
type: integer
items:
type: array
items:
type: object
properties:
uuid:
type: string
format: uuid
url:
type: string
format: uri
jobId:
type:
- string
- 'null'
status:
type: string
enum:
- pending
- running
- completed
- failed
- cancelled
result:
description: >-
Extracted content. Object for JSON output, string
for markdown. null until completed.
error:
type:
- string
- 'null'
creditsUsed:
type: number
startedAt:
type:
- string
- 'null'
format: date-time
finishedAt:
type:
- string
- 'null'
format: date-time
screenshotUrl:
type:
- string
- 'null'
format: uri
createdAt:
type: string
format: date-time
finishedAt:
type:
- string
- 'null'
format: date-time
'401':
description: Missing x-api-key header
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
'404':
description: Batch not found or belongs to a different user
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
example:
status: error
message: Batch scrape job not found.
code: NOT_FOUND
security:
- ApiKeyAuth: []
components:
schemas:
ErrorResponse:
type: object
properties:
status:
type: string
enum:
- error
message:
type: string
required:
- status
- message
securitySchemes:
ApiKeyAuth:
type: apiKey
in: header
name: x-api-key
````