Node

Scrape pages, run browser actions, batch-process URLs, and crawl entire sites. All results come back as structured data ready to feed into your LLM pipelines or store directly.

Installation

To install the Spidra Node SDK, you can use npm:

npm install spidra-js

Get your API key from app.spidra.io under Settings → API Keys. Never hardcode it in source files — use an environment variable instead.

Setup

Here’s an example of initializing the Spidra client in a Node.js or TypeScript project.

import { SpidraClient } from 'spidra-js';

const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY });

Scraping

All scrape jobs run asynchronously. run() submits a job and polls until it finishes. For manual control, use submit() and get() directly. Up to 3 URLs can be passed per request and are processed in parallel.

Scrape a web page

Submit a scrape job and wait for results.

const job = await spidra.scrape.run({
	urls: [{ url: 'https://example.com/pricing' }],
	prompt: 'Extract all pricing plans with name, price, and included features',
	output: 'json',
});

console.log(job.result.content);
// { plans: [{ name: "Starter", price: "$9/mo", features: [...] }, ...] }

Parameters

Parameter	Type	Description
`urls`	`{ url: string, actions?: Action[] }[]`	URLs to scrape, with optional per-URL browser actions
`prompt`	`string`	AI extraction instruction
`output`	`"markdown"` \| `"json"`	Response format. Defaults to `"markdown"`
`schema`	`object`	JSON Schema for guaranteed output shape (use with `output: "json"`)
`useProxy`	`boolean`	Route through a residential proxy
`proxyCountry`	`string`	Two-letter country code, e.g. `"us"`, `"de"`, `"jp"`
`extractContentOnly`	`boolean`	Strip navigation, ads, and boilerplate before AI extraction
`screenshot`	`boolean`	Capture a screenshot of the page
`fullPageScreenshot`	`boolean`	Capture a full-page (scrolled) screenshot
`cookies`	`string`	Raw `Cookie` header string for authenticated pages

Fire-and-forget approach

Fire-and-forget approach: submit a job immediately and poll on your own schedule.

// Submit — returns immediately with a jobId
const { jobId } = await spidra.scrape.submit({
	urls: [{ url: 'https://example.com' }],
	prompt: 'Extract the main headline',
});

// Check status at any time
const status = await spidra.scrape.get(jobId);

if (status.status === 'completed') {
	console.log(status.result.content);
} else if (status.status === 'failed') {
	console.error(status.error);
}

Job statuses: waiting · active · completed · failed

Structured JSON output

Pass a schema to enforce an exact output shape. Missing fields come back as null rather than hallucinated values.

const job = await spidra.scrape.run({
	urls: [{ url: 'https://jobs.example.com/senior-engineer' }],
	prompt: 'Extract the job listing details',
	output: 'json',
	schema: {
		type: 'object',
		required: ['title', 'company', 'remote'],
		properties: {
			title: { type: 'string' },
			company: { type: 'string' },
			remote: { type: ['boolean', 'null'] },
			salary_min: { type: ['number', 'null'] },
			salary_max: { type: ['number', 'null'] },
			skills: { type: 'array', items: { type: 'string' } },
		},
	},
});

Geo-targeted scraping

Route through a residential proxy in a specific country for geo-restricted content or localized pricing.

const job = await spidra.scrape.run({
	urls: [{ url: 'https://www.amazon.de/gp/bestsellers' }],
	prompt: 'List the top 10 products with name and price',
	useProxy: true,
	proxyCountry: 'de',
});

Supported codes include us, gb, de, fr, jp, au, ca, br, in, nl, sg, es, it, mx, and 40+ more. Use "global" or "eu" for regional routing.

Authenticated pages

Pass session cookies as a raw header string to scrape pages behind a login.

const job = await spidra.scrape.run({
	urls: [{ url: 'https://app.example.com/dashboard' }],
	prompt: 'Extract the monthly revenue and active user count',
	cookies: 'session=abc123; auth_token=xyz789',
});

Browser actions

Run actions against the page before extraction. They execute in order — the scrape happens after all actions complete.

const job = await spidra.scrape.run({
	urls: [
		{
			url: 'https://example.com/products',
			actions: [
				{ type: 'click', selector: '#accept-cookies' },
				{ type: 'wait', duration: 1000 },
				{ type: 'scroll', to: '80%' },
			],
		},
	],
	prompt: 'Extract all product names and prices',
});

Available actions

Action	Required fields	Description
`click`	`selector` or `value`	Click a button, link, or any element
`type`	`selector`, `value`	Type text into an input or textarea
`check`	`selector` or `value`	Check a checkbox
`uncheck`	`selector` or `value`	Uncheck a checkbox
`wait`	`duration` (ms)	Pause for a set number of milliseconds
`scroll`	`to` (`0–100%`)	Scroll the page to a percentage of its height
`forEach`	`observe`	Loop over every matched element and process each one

Use selector for a CSS selector or XPath. Use value for plain English — Spidra locates the element using AI.

{ type: 'click', selector: "button[data-testid='submit']" }   // CSS selector
{ type: 'click', value: 'Accept all cookies button' }          // plain English
{ type: 'type',  selector: "input[name='q']", value: 'wireless headphones' }
{ type: 'wait',  duration: 2000 }
{ type: 'scroll', to: '100%' }

forEach — loop over every element

forEach finds a set of matching elements on the page and processes each one individually. Use it when you need to collect data from a list of items, paginate across pages, or click into each item’s detail page.

You don’t need forEach if all the data fits on a single page — a plain prompt is simpler and works just as well.

Use forEach when:

The list spans multiple pages and you need pagination
You need to click into each item’s detail page (navigate mode)
You have 20+ items and want consistent per-item AI extraction (itemPrompt)

inline mode

Read each element’s content directly without navigating away. Best for product cards, search results, and table rows.

const job = await spidra.scrape.run({
	urls: [
		{
			url: 'https://books.toscrape.com',
			actions: [
				{
					type: 'forEach',
					observe: 'Find all book cards in the product grid',
					mode: 'inline',
					captureSelector: 'article.product_pod',
					maxItems: 20,
					itemPrompt:
						'Extract title, price, and star rating as JSON: {title, price, star_rating}',
				},
			],
		},
	],
	prompt: 'Return a clean JSON array of all books',
	output: 'json',
});

navigate mode

Follow each element’s link to its destination page and capture content there. Best for product listings where full details are only on individual pages.

{
  type:            'forEach',
  observe:         'Find all book title links in the product grid',
  mode:            'navigate',
  captureSelector: 'article.product_page',
  maxItems:        10,
  waitAfterClick:  800,
  itemPrompt:      'Extract title, price, star rating, and availability as JSON',
}

click mode

Click each element, capture the content that appears (modal, drawer, or expanded section), then move on. Best for hotel room cards, FAQ accordions, or any UI where clicking reveals hidden content.

{
  type:            'forEach',
  observe:         'Find all room type cards',
  mode:            'click',
  captureSelector: "[role='dialog']",
  maxItems:        8,
  waitAfterClick:  1200,
  itemPrompt:      'Extract room name, bed type, price per night, and amenities as JSON',
}

Pagination

After processing all elements on the current page, follow the next-page link and continue.

{
  type:     'forEach',
  observe:  'Find all book title links',
  mode:     'navigate',
  maxItems: 40,
  pagination: {
    nextSelector: 'li.next > a',
    maxPages:     3,   // 3 additional pages beyond the first
  },
}

maxItems applies across all pages combined. The loop stops when you hit maxItems, run out of elements, or reach maxPages.

Per-element actions

Run extra browser actions on each item after navigating or clicking into it, before content is captured. Useful for scrolling below the fold or expanding collapsed sections.

{
  type:            'forEach',
  observe:         'Find all book title links',
  mode:            'navigate',
  captureSelector: 'article.product_page',
  maxItems:        5,
  waitAfterClick:  1000,
  actions: [
    { type: 'scroll', to: '50%' },
  ],
  itemPrompt: 'Extract title, price, and full description as JSON',
}

itemPrompt vs top-level prompt

Both are optional and serve different purposes:

	`itemPrompt`	`prompt`
Runs	During scraping, once per item	After all items are collected
Sees	One item’s content	All items combined
Output	`result.data[].markdownContent`	`result.content`

Use itemPrompt to extract fields from each item individually. Use the top-level prompt to filter, sort, or reshape the combined output. They can be used together.

Poll options

scrape.run(), batch.run(), and crawl.run() accept a second argument to control polling behavior.

const job = await spidra.scrape.run(params, {
	pollInterval: 3000, // ms between status checks (default: 3000)
	timeout: 120_000, // max wait in ms before throwing (default: 120000)
});

Batch scraping

Submit up to 50 URLs in one request. All URLs are processed in parallel. Each URL is a plain string (not an object).

batch.run()

const batch = await spidra.batch.run({
	urls: [
		'https://shop.example.com/product/1',
		'https://shop.example.com/product/2',
		'https://shop.example.com/product/3',
	],
	prompt: 'Extract product name, price, and availability',
	output: 'json',
	useProxy: true,
});

console.log(`${batch.completedCount}/${batch.totalUrls} succeeded`);

for (const item of batch.items) {
	if (item.status === 'completed') console.log(item.url, item.result);
	if (item.status === 'failed') console.error(item.url, item.error);
}

Item statuses: pending · running · completed · failed Batch statuses: pending · running · completed · failed · cancelled

batch.submit() + batch.get()

const { batchId } = await spidra.batch.submit({
	urls: ['https://example.com/1', 'https://example.com/2'],
	prompt: 'Extract the page title',
});

const result = await spidra.batch.get(batchId);
console.log(result.status, result.completedCount, '/', result.totalUrls);

Retry failed items

Re-queue only the items that failed — successful items are not re-run.

const result = await spidra.batch.get(batchId);

if (result.failedCount > 0) {
	const { retriedCount } = await spidra.batch.retry(batchId);
	console.log(`Retrying ${retriedCount} items`);
}

Cancel a batch

Stops all pending items and refunds credits for unprocessed work.

const { cancelledItems, creditsRefunded } = await spidra.batch.cancel(batchId);
console.log(
	`Cancelled ${cancelledItems} items, refunded ${creditsRefunded} credits`,
);

List past batches

const { jobs, pagination } = await spidra.batch.list({ page: 1, limit: 20 });

for (const job of jobs) {
	console.log(job.uuid, job.status, `${job.completedCount}/${job.totalUrls}`);
}

Crawling

Give Spidra a starting URL and instructions for which links to follow. It discovers pages automatically, extracts structured data from each one, and returns everything when the crawl is done.

crawl.run()

const job = await spidra.crawl.run({
	baseUrl: 'https://competitor.com/blog',
	crawlInstruction: 'Follow blog post links only, skip tag and category pages',
	transformInstruction:
		'Extract the title, author, publish date, and a one-sentence summary',
	maxPages: 30,
	useProxy: true,
});

for (const page of job.result) {
	console.log(page.url, page.data);
}

Parameters

Parameter	Type	Description
`baseUrl`	`string`	Starting URL for the crawl
`crawlInstruction`	`string`	Which links to follow and which to skip
`transformInstruction`	`string`	What to extract from each page
`maxPages`	`number`	Maximum number of pages to crawl
`useProxy`	`boolean`	Route through a residential proxy
`proxyCountry`	`string`	Two-letter country code, e.g. `"us"`
`cookies`	`string`	Raw `Cookie` header string for authenticated sites

crawl.submit() + crawl.get()

const { jobId } = await spidra.crawl.submit({
	baseUrl: 'https://example.com/docs',
	crawlInstruction: 'Find all documentation pages',
	transformInstruction: 'Extract the page title and main content summary',
	maxPages: 50,
});

// Poll manually
const status = await spidra.crawl.get(jobId);
// status: "waiting" | "active" | "running" | "completed" | "failed"

crawl.pages() — download crawled content

Returns signed S3 URLs for the raw HTML and Markdown of each crawled page. Links expire after 1 hour.

const { pages } = await spidra.crawl.pages(jobId);

for (const page of pages) {
	console.log(page.url, page.status);
	// page.html_url     — download raw HTML
	// page.markdown_url — download Markdown version
}

crawl.extract() — re-extract without re-crawling

Apply a new AI prompt to an existing completed crawl without fetching the pages again. Only transformation credits are charged.

const { jobId: newJobId } = await spidra.crawl.extract(
	sourceJobId,
	'Extract only the product SKUs and prices as a CSV',
);

// Poll the new extraction job
const result = await spidra.crawl.get(newJobId);

History and stats

// List past crawl jobs
const { jobs, total, page, totalPages } = await spidra.crawl.history({
	page: 1,
	limit: 10,
});

// Total crawl job count for your account
const { total: totalCrawls } = await spidra.crawl.stats();

Logs

Every API scrape job is logged automatically. Access your full history with optional filters.

logs.list()

const { logs, total } = await spidra.logs.list({
	status: 'failed', // "success" | "failed"
	searchTerm: 'amazon.com',
	channel: 'api', // "api" | "playground"
	dateStart: '2024-01-01',
	dateEnd: '2024-12-31',
	page: 1,
	limit: 20,
});

for (const log of logs) {
	console.log(log.urls[0]?.url, log.status, log.credits_used);
}

Filter parameters

Parameter	Type	Description
`status`	`"success"` \| `"failed"`	Filter by outcome
`searchTerm`	`string`	Search by URL or prompt
`channel`	`string`	`"api"` or `"playground"`
`dateStart`	`string`	ISO date — return logs on or after this date
`dateEnd`	`string`	ISO date — return logs on or before this date
`page`	`number`	Page number (default: 1)
`limit`	`number`	Results per page (default: 20)

logs.get()

Get a single log entry including the full AI extraction result.

const log = await spidra.logs.get(logUuid);
console.log(log.result_data); // full AI output for that job

Usage statistics

Returns credit and request usage broken down by day or week.

// Range options: "7d" | "30d" | "weekly"
const rows = await spidra.usage.get('30d');

for (const row of rows) {
	console.log(row.date, row.requests, row.credits);
}

Range	Description
`"7d"`	Last 7 days, one row per day
`"30d"`	Last 30 days, one row per day
`"weekly"`	Last 7 weeks, one row per week

Error handling

Every non-2xx response throws a typed error class. Catch the specific class you care about, or fall back to the base SpidraError.

import {
	SpidraClient,
	SpidraError,
	SpidraAuthenticationError,
	SpidraInsufficientCreditsError,
	SpidraRateLimitError,
	SpidraServerError,
} from 'spidra-js';

try {
	await spidra.scrape.run({
		urls: [{ url: 'https://example.com' }],
		prompt: '...',
	});
} catch (err) {
	if (err instanceof SpidraAuthenticationError) {
		console.error('Invalid or missing API key'); // 401
	} else if (err instanceof SpidraInsufficientCreditsError) {
		console.error('Out of credits — top up your account'); // 403
	} else if (err instanceof SpidraRateLimitError) {
		console.error('Rate limited — back off and retry'); // 429
	} else if (err instanceof SpidraServerError) {
		console.error('Server error — try again shortly'); // 500
	} else if (err instanceof SpidraError) {
		console.error(`API error ${err.status}: ${err.message}`);
	}
}

Error classes

Class	Status	When
`SpidraAuthenticationError`	401	Missing or invalid `x-api-key` header
`SpidraInsufficientCreditsError`	403	Account has no remaining credits
`SpidraRateLimitError`	429	Too many requests
`SpidraServerError`	500	Unexpected error on Spidra’s side
`SpidraError`	other	Any other non-2xx response

All error classes expose err.status (HTTP status code) and err.message.

AI agent integration

Spidra works as a tool inside AI agent pipelines. Here is an example using the Vercel AI SDK with Claude:

import { generateText, tool } from 'ai';
import { anthropic } from '@ai-sdk/anthropic';
import { SpidraClient } from 'spidra-js';
import { z } from 'zod';

const spidra = new SpidraClient({ apiKey: process.env.SPIDRA_API_KEY });

const result = await generateText({
	model: anthropic('claude-opus-4-6'),
	maxSteps: 5,
	tools: {
		scrapeUrl: tool({
			description: 'Fetch and extract structured data from a URL',
			parameters: z.object({
				url: z.string().describe('The URL to scrape'),
				prompt: z.string().describe('What data to extract'),
			}),
			execute: async ({ url, prompt }) => {
				const job = await spidra.scrape.run({ urls: [{ url }], prompt });
				return JSON.stringify(job.result.content);
			},
		}),
	},
	prompt: 'What are the top 3 trending repositories on GitHub today?',
});

console.log(result.text);

Overview

Official SDKs

Installation

Setup

Scraping

Scrape a web page

Fire-and-forget approach

Structured JSON output

Geo-targeted scraping

Authenticated pages

Browser actions

forEach — loop over every element

inline mode

navigate mode

click mode

Per-element actions

itemPrompt vs top-level prompt

Poll options

Batch scraping

batch.run()

batch.submit() + batch.get()

Retry failed items

Cancel a batch

List past batches

Crawling

crawl.run()

crawl.submit() + crawl.get()

crawl.pages() — download crawled content

crawl.extract() — re-extract without re-crawling

History and stats

Logs

logs.list()

logs.get()

Usage statistics

Error handling

AI agent integration

Overview

Official SDKs

​Installation

​Setup

​Scraping

​Scrape a web page

​Fire-and-forget approach

​Structured JSON output

​Geo-targeted scraping

​Authenticated pages

​Browser actions

​forEach — loop over every element

​inline mode

​navigate mode

​click mode

​Pagination

​Per-element actions

​itemPrompt vs top-level prompt

​Poll options

​Batch scraping

​batch.run()

​batch.submit() + batch.get()

​Retry failed items

​Cancel a batch

​List past batches

​Crawling

​crawl.run()

​crawl.submit() + crawl.get()

​crawl.pages() — download crawled content

​crawl.extract() — re-extract without re-crawling

​History and stats

​Logs

​logs.list()

​logs.get()

​Usage statistics

​Error handling

​AI agent integration

Installation

Setup

Scraping

Scrape a web page

Fire-and-forget approach

Structured JSON output

Geo-targeted scraping

Authenticated pages

Browser actions

forEach — loop over every element

inline mode

navigate mode

click mode

Pagination

Per-element actions

itemPrompt vs top-level prompt

Poll options

Batch scraping

batch.run()

batch.submit() + batch.get()

Retry failed items

Cancel a batch

List past batches

Crawling

crawl.run()

crawl.submit() + crawl.get()

crawl.pages() — download crawled content

crawl.extract() — re-extract without re-crawling

History and stats

Logs

logs.list()

logs.get()

Usage statistics

Error handling

AI agent integration