> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Next.js

> Use Spidra's web scraping, site crawling, SERP search, and document parsing in Next.js with React Server Components, Server Actions, Route Handlers, and HMAC webhooks.

The Spidra Next.js integration (`spidra-nextjs`) brings Spidra's web intelligence platform natively to Next.js (App Router, Pages Router, and Server Actions). It provides cached client accessors for React Server Components (`getSpidraClient()`), type-safe wrappers for Server Actions (`createSpidraAction()`), pre-built Route Handlers for `/api/spidra/*`, and HMAC-SHA256 webhook handlers.

## Installation

```bash theme={null}
npm install spidra-nextjs spidra
```

Requires Node.js 18+ and Next.js 13+ (fully compatible with Next.js 14 and 15).

> [!NOTE]
>
> Get your API key from [app.spidra.io](https://app.spidra.io) under **API Keys**.
> Store it in `.env.local`. Never commit API keys to version control.

## Configuration

Configure credentials in `.env.local`:

```bash theme={null}
SPIDRA_API_KEY=spd_live_your_api_key
SPIDRA_WEBHOOK_SECRET=whsec_your_signing_secret

# Optional overrides
SPIDRA_BASE_URL=https://api.spidra.io/api
SPIDRA_MAX_RETRIES=3
SPIDRA_BACKOFF_FACTOR=1.0
```

`getSpidraClient()` automatically reads `SPIDRA_API_KEY` and optional overrides from your environment.

## Integration patterns

### 1. React Server Components (RSC)

Call Spidra directly inside async Server Components without client-side waterfalls:

```tsx theme={null}
// app/products/page.tsx
import { getSpidraClient } from "spidra-nextjs";

export default async function ProductsPage() {
  const spidra = getSpidraClient();

  const result = await spidra.scrape({
    urls: [{ url: "https://books.toscrape.com" }],
    prompt: "Extract the top 5 books with title, price, and rating",
    output: "json",
  });

  const books = result.data?.[0]?.content?.books ?? [];

  return (
    <main className="p-8 max-w-4xl mx-auto">
      <h1 className="text-2xl font-bold mb-4">Book Listings</h1>
      <div className="grid gap-4">
        {books.map((book: any, idx: number) => (
          <div key={idx} className="border p-4 rounded shadow-sm">
            <h2 className="font-semibold text-lg">{book.title}</h2>
            <p className="text-emerald-600 font-medium">{book.price}</p>
          </div>
        ))}
      </div>
    </main>
  );
}
```

### 2. Server Actions

Use `createSpidraAction()` to expose type-safe scraping functions callable directly from Client Components:

```ts theme={null}
// app/actions/scrape.ts
"use server";

import { createSpidraAction } from "spidra-nextjs";

export const scrapeUrlAction = createSpidraAction(async (client, targetUrl: string) => {
  return await client.scrape({
    urls: [{ url: targetUrl }],
    prompt: "Extract page headline and a 3-bullet summary",
    output: "json",
  });
});
```

Call the action from an interactive form:

```tsx theme={null}
// app/components/ScraperForm.tsx
"use client";

import { useState } from "react";
import { scrapeUrlAction } from "@/app/actions/scrape";

export function ScraperForm() {
  const [url, setUrl] = useState("");
  const [loading, setLoading] = useState(false);
  const [data, setData] = useState<any>(null);

  async function handleScrape(e: React.FormEvent) {
    e.preventDefault();
    setLoading(true);
    const res = await scrapeUrlAction(url);
    setLoading(false);
    if (res.success) {
      setData(res.data);
    } else {
      alert(`Error: ${res.error}`);
    }
  }

  return (
    <form onSubmit={handleScrape} className="space-y-4">
      <input
        type="url"
        value={url}
        onChange={(e) => setUrl(e.target.value)}
        placeholder="https://example.com"
        className="w-full border p-2 rounded"
        required
      />
      <button type="submit" disabled={loading} className="bg-blue-600 text-white px-4 py-2 rounded">
        {loading ? "Extracting..." : "Extract Data"}
      </button>
      {data && <pre className="bg-gray-100 p-4 rounded mt-4">{JSON.stringify(data, null, 2)}</pre>}
    </form>
  );
}
```

### 3. Pre-built Route Handlers

Mount complete Spidra REST API endpoints under `/api/spidra/[...action]` in one line:

```ts theme={null}
// app/api/spidra/[...action]/route.ts
import { createSpidraRouteHandler } from "spidra-nextjs";

export const { GET, POST } = createSpidraRouteHandler();
```

This immediately exposes:

* `POST /api/spidra/scrape`
* `GET/POST /api/spidra/search`
* `POST /api/spidra/crawl`
* `POST /api/spidra/batch`
* `POST /api/spidra/map`
* `POST /api/spidra/parse`
* `GET /api/spidra/account`
* `GET /api/spidra/usage`

### 4. HMAC Webhooks

Verify incoming webhook deliveries (`crawl.page`, `crawl.completed`, `batch.completed`) with `createSpidraWebhookHandler()`:

```ts theme={null}
// app/api/webhooks/spidra/route.ts
import { createSpidraWebhookHandler } from "spidra-nextjs";

export const POST = createSpidraWebhookHandler({
  onEvent: async (event) => {
    if (event.event === "crawl.completed") {
      console.log(`Crawl job ${event.jobId} completed!`);
      // Update database or trigger revalidation
    } else if (event.event === "crawl.page") {
      console.log("New page extracted:", event.page?.url);
    }
  },
});
```

The handler automatically verifies the `X-Spidra-Signature` header against `process.env.SPIDRA_WEBHOOK_SECRET` in constant time.

## Scraping & browser actions

Run browser automation steps (typing, clicking, scrolling, waiting) before extraction:

```ts theme={null}
import { getSpidraClient } from "spidra-nextjs";

const spidra = getSpidraClient();

const result = await spidra.scrape({
  urls: [
    {
      url: "https://quotes.toscrape.com/login",
      actions: [
        { type: "type", selector: "#username", value: "admin" },
        { type: "type", selector: "#password", value: "admin" },
        { type: "click", selector: "input[type='submit']" },
        { type: "waitFor", selector: "a[href='/logout']" },
      ],
    },
  ],
  prompt: "Extract the top 5 quotes with author",
  output: "json",
  useProxy: true,
  proxyCountry: "us",
});
```

## Deep site crawling

Recursively crawl domains with path filtering:

```ts theme={null}
const crawlJob = await spidra.crawl({
  baseUrl: "https://docs.example.com",
  crawlInstruction: "Follow documentation guides, skip release notes and changelogs",
  transformInstruction: "Extract the page title and key code snippets",
  maxPages: 30,
  includePaths: ["/guides/*", "/tutorials/*"],
});
```

## Live SERP & web search

Search the web and extract results across multiple sources:

```ts theme={null}
const results = await spidra.search({
  query: "Next.js App Router performance optimizations",
  sources: ["web", "news"],
  limit: 10,
  timeRange: "week",
});
```

## Batch scraping & streaming watchers

Submit up to 50 URLs concurrently and stream items as they finish:

```ts theme={null}
const batch = await spidra.startBatchScrape({
  urls: [
    "https://example.com/page-1",
    "https://example.com/page-2",
    "https://example.com/page-3",
  ],
  prompt: "Extract title and meta description",
});

const watcher = spidra.watchBatch(batch.batchId);
watcher.on("item", (item) => {
  console.log(`Finished ${item.url}:`, item.status);
});
```

## Domain mapping

```ts theme={null}
const map = await spidra.map({
  url: "https://example.com",
  limit: 100,
});
```

## Document parsing (PDF, Word, Excel)

```ts theme={null}
import { readFile } from "fs/promises";

const pdfBytes = await readFile("./contract.pdf");

const parsed = await spidra.parse({
  file: pdfBytes,
  fileName: "contract.pdf",
}, {
  prompt: "Extract effective date, parties, and total agreement amount",
});
```

## Ready-made templates & monitors

```ts theme={null}
// Run Google Maps leads template
const run = await spidra.runTemplate("google-maps-leads", {
  query: "plumbers",
  areas: ["Austin, TX"],
});

// Setup automated change monitor
const monitor = await spidra.createMonitor({
  name: "Pricing Watcher",
  sources: [{ kind: "urls", urls: ["https://example.com/pricing"] }],
  schedule: { every: "1d", at: "09:00" },
  alertOn: "page",
  goal: "Notify if pricing amounts change",
  notify: {
    webhook: { url: "https://my-app.com/api/webhooks/spidra" },
  },
});
```

## Error handling

```ts theme={null}
import {
  SpidraAuthenticationError,
  SpidraRateLimitError,
  SpidraPaymentRequiredError,
  SpidraError,
} from "spidra-nextjs";

try {
  await spidra.scrape({ urls: [{ url: "https://example.com" }] });
} catch (err) {
  if (err instanceof SpidraAuthenticationError) {
    console.error("Invalid API Key");
  } else if (err instanceof SpidraPaymentRequiredError) {
    console.error("Credits exhausted");
  } else if (err instanceof SpidraRateLimitError) {
    console.warn(`Rate limited. Retry after ${err.retryAfter}s`);
  } else if (err instanceof SpidraError) {
    console.error(`Spidra error (${err.status}): ${err.message}`);
  }
}
```

> [card] [Node.js SDK](https://docs.spidra.io/sdks/node.md)
>
> Core Node.js / TypeScript SDK documentation and API reference.

> [card] [Express](https://docs.spidra.io/sdks/express.md)
>
> Express.js middleware, turnkey router, and error handling.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.