Re-Extract from Existing Crawl
Crawl Endpoints
Extract from Crawl
Run a new extraction on pages from a completed crawl job without re-crawling the site
POST
Re-Extract from Existing Crawl
Prerequisites
Request Body
How It Works
- Pass the
jobIdof a completed crawl job. If you ran the crawl previously, this is theidfield shown in your crawl history. - Provide a
transformInstructiondescribing what you want to extract. - Spidra loads the saved content for each page and runs your prompt against it.
- A new crawl job is created with the results, which you can poll and download the same way as any other job.
When to Use This
- You want to extract different fields from pages you already crawled
- Your first extraction prompt wasn’t quite right and you want to try again
- You need the same pages in two different formats, like JSON and CSV
Polling Results
The response returns a newjobId. Use the standard crawl endpoints to check progress and get results:
Common Errors
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Path Parameters
The ID of the completed source crawl job to extract from
Body
application/json
Extraction prompt to apply to all pages from the source crawl. Maximum 5,000 characters.
Maximum string length:
5000
