Fastify

The Spidra Fastify integration (spidra-fastify) provides a first-class plugin for Fastify applications:

spidraPlugin: Decorates fastify.spidra and request.spidra with the official client.
Turnkey Routes: Optional routes: true registers /spidra/scrape, /spidra/crawl, /spidra/search, /spidra/batch, /spidra/map, /spidra/parse, /spidra/account, and /spidra/usage automatically.
HMAC Webhook Verification: Built-in verification for X-Spidra-Signature on asynchronous crawl and batch events.

Installation

npm install spidra-fastify spidra

Requires Node.js 18+ and Fastify 4+.

Get your API key from app.spidra.io under API Keys. Store it in your environment variables.

Configuration

Set environment variables in .env:

SPIDRA_API_KEY=spd_live_your_api_key
SPIDRA_WEBHOOK_SECRET=whsec_your_signing_secret

# Optional overrides
SPIDRA_BASE_URL=https://api.spidra.io/api
SPIDRA_MAX_RETRIES=3
SPIDRA_BACKOFF_FACTOR=1.0

Quickstart

1. Register plugin & decorate instance

import Fastify from "fastify";
import spidraPlugin from "spidra-fastify";

const fastify = Fastify({ logger: true });

await fastify.register(spidraPlugin, {
  apiKey: process.env.SPIDRA_API_KEY,
});

fastify.post("/scrape", async (request, reply) => {
  const { url } = request.body as any;

  const result = await request.spidra.scrape({
    urls: [{ url }],
    prompt: "Extract page headline and key takeaways",
    output: "markdown",
  });

  return result;
});

await fastify.listen({ port: 3000 });

2. Turnkey REST endpoints

Mount all Spidra REST API endpoints under /api/spidra in one line:

await fastify.register(spidraPlugin, {
  apiKey: process.env.SPIDRA_API_KEY,
  routes: true,
  prefix: "/api/spidra",
  webhookSecret: process.env.SPIDRA_WEBHOOK_SECRET,
  onWebhookEvent: async (event) => {
    console.log("Verified Spidra event:", event.event);
  },
});

Scraping & browser actions

fastify.post("/scrape-quotes", async (request, reply) => {
  const result = await request.spidra.scrape({
    urls: [
      {
        url: "https://quotes.toscrape.com/login",
        actions: [
          { type: "type", selector: "#username", value: "admin" },
          { type: "type", selector: "#password", value: "admin" },
          { type: "click", selector: "input[type='submit']" },
          { type: "waitFor", selector: "a[href='/logout']" },
        ],
      },
    ],
    prompt: "Extract top 5 quotes with author",
    output: "json",
    useProxy: true,
    proxyCountry: "us",
  });

  return result;
});

Deep site crawling

fastify.post("/crawl-site", async (request, reply) => {
  const { url } = request.body as any;

  const crawlJob = await request.spidra.crawl({
    baseUrl: url,
    crawlInstruction: "Follow documentation pages only",
    transformInstruction: "Extract title and code snippets",
    maxPages: 30,
    includePaths: ["/docs/*"],
  });

  return { pages: crawlJob.result };
});

Live SERP & web search

fastify.get("/search", async (request, reply) => {
  const { q } = request.query as any;

  const results = await request.spidra.search({
    query: q,
    sources: ["web", "news"],
    limit: 10,
  });

  return results;
});

Batch scraping

fastify.post("/batch", async (request, reply) => {
  const { urls } = request.body as any;

  const batch = await request.spidra.batchScrape({
    urls,
    prompt: "Extract title and description",
    output: "json",
  });

  return batch;
});

Domain mapping

fastify.get("/map", async (request, reply) => {
  const { url } = request.query as any;
  const map = await request.spidra.map({ url, limit: 100 });
  return map;
});

Document parsing (PDF, Word, Excel)

import { readFile } from "fs/promises";

fastify.post("/parse-document", async (request, reply) => {
  const bytes = await readFile("./report.pdf");

  const doc = await request.spidra.parse({
    file: bytes,
    fileName: "report.pdf",
  }, {
    prompt: "Extract key figures and dates",
  });

  return { markdown: doc.markdown, data: doc.json };
});

Turnkey templates & monitors

fastify.post("/leads", async (request, reply) => {
  const { query, area } = request.body as any;

  const run = await request.spidra.runTemplate("google-maps-leads", {
    query,
    areas: [area],
  });

  return run.rows;
});

fastify.post("/monitors", async (request, reply) => {
  const { url, webhookUrl } = request.body as any;

  const monitor = await request.spidra.createMonitor({
    name: "Price Watcher",
    sources: [{ kind: "urls", urls: [url] }],
    schedule: { every: "1d", at: "09:00" },
    alertOn: "page",
    goal: "Alert on any price change",
    notify: { webhook: { url: webhookUrl } },
  });

  return monitor;
});

Error handling

import {
  SpidraAuthenticationError,
  SpidraRateLimitError,
  SpidraPaymentRequiredError,
  SpidraError,
} from "spidra-fastify";

fastify.setErrorHandler((error, request, reply) => {
  if (error instanceof SpidraAuthenticationError) {
    reply.status(401).send({ error: "Invalid Spidra API Key" });
  } else if (error instanceof SpidraPaymentRequiredError) {
    reply.status(402).send({ error: "Spidra credit balance exhausted" });
  } else if (error instanceof SpidraRateLimitError) {
    reply.status(429).send({ error: "Rate limited", retryAfter: error.retryAfter });
  } else if (error instanceof SpidraError) {
    reply.status(error.status || 500).send({ error: error.message, code: error.code });
  } else {
    reply.send(error);
  }
});

NestJS

NestJS dynamic modules, injectable service, and webhook guard.

Hono

Cloudflare Workers and edge-ready web standard middleware.

SDKs

AI tools

Installation

Configuration

Quickstart

1. Register plugin & decorate instance

2. Turnkey REST endpoints

Scraping & browser actions

Deep site crawling

Live SERP & web search

Batch scraping

Domain mapping

Document parsing (PDF, Word, Excel)

Turnkey templates & monitors

Error handling

NestJS

Hono