> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Remix

> Seamless web scraping, site crawling, SERP search, and document parsing for Remix and React Router v7 applications.

The Spidra Remix integration (`spidra-remix`) brings web data intelligence to fullstack Remix and React Router v7 applications:

* **Loaders & Actions**: `createSpidraLoader()` and `createSpidraAction()` generators with automatic typed error responses.
* **HMAC Webhook Handler**: `handleRemixWebhook()` validates asynchronous job events and dispatches typed callbacks.
* **Remix Response Helper**: `spidraJson()` produces standard Remix `Response` objects with proper HTTP status codes.
* **Full API Surface**: Scrape, Crawl, SERP Search, Batch Scraping, Document Parsing, Domain Mapping, Turnkey Templates, Presets, and Monitors.

## Installation

```bash theme={null}
npm install spidra-remix spidra
```

> [!NOTE]
>
> Get your API key from [app.spidra.io](https://app.spidra.io) under **API Keys**.
> Store it in `.env` or set `SPIDRA_API_KEY`.

## Configuration

Set environment variables in `.env`:

```bash theme={null}
SPIDRA_API_KEY=spd_live_your_api_key
SPIDRA_WEBHOOK_SECRET=whsec_your_signing_secret

# Optional overrides
SPIDRA_BASE_URL=https://api.spidra.io/api
SPIDRA_MAX_RETRIES=3
SPIDRA_BACKOFF_FACTOR=1.0
```

`getSpidraClient()` automatically reads `SPIDRA_API_KEY` from `process.env`.

## Quickstart

### 1. In a Remix Action

Handle form submissions and extract structured data:

```ts theme={null}
// app/routes/scrape.tsx
import type { ActionFunctionArgs } from "@remix-run/node";
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const formData = await request.formData();
  const url = formData.get("url") as string;

  return await client.scrape({
    urls: [{ url }],
    prompt: "Extract product title, price, and customer review rating",
    output: "json",
  });
});
```

### 2. In a Remix Loader

Pre-fetch search or crawl data on the server before rendering:

```ts theme={null}
// app/routes/search.tsx
import type { LoaderFunctionArgs } from "@remix-run/node";
import { createSpidraLoader } from "spidra-remix";

export const loader = createSpidraLoader(async ({ client, request }) => {
  const url = new URL(request.url);
  const q = url.searchParams.get("q") || "Remix fullstack development";

  return await client.search({
    query: q,
    sources: ["web", "news"],
    limit: 10,
  });
});
```

### 3. Asynchronous Webhooks

Handle asynchronous crawl or batch job completions securely:

```ts theme={null}
// app/routes/api.webhooks.spidra.ts
import type { ActionFunctionArgs } from "@remix-run/node";
import { handleRemixWebhook } from "spidra-remix";

export const action = async ({ request }: ActionFunctionArgs) => {
  return await handleRemixWebhook({
    request,
    webhookSecret: process.env.SPIDRA_WEBHOOK_SECRET,
    onEvent: async (event) => {
      console.log(`Received verified webhook event: ${event.event}`);

      if (event.event === "crawl.completed") {
        console.log(`Crawl job ${event.jobId} completed!`);
      }
    },
  });
};
```

## Complete Spidra Features Guide

### 1. Scraping with Browser Actions & Proxies

Bypass anti-bot mechanisms with residential proxies and interact with dynamic pages before extraction:

```ts theme={null}
// app/routes/api.scrape-dynamic.ts
import type { ActionFunctionArgs } from "@remix-run/node";
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { targetUrl } = await request.json();

  const result = await client.scrape({
    urls: [{ url: targetUrl }],
    prompt: "Extract interactive table items and prices",
    output: "json",
    schema: {
      type: "object",
      properties: {
        items: {
          type: "array",
          items: {
            type: "object",
            properties: {
              title: { type: "string" },
              price: { type: "number" },
            },
          },
        },
      },
    },
    actions: [
      { type: "click", selector: "button#load-more" },
      { type: "wait", ms: 2000 },
      { type: "scroll", distance: 1000 },
    ],
    proxy: {
      type: "residential",
      country: "US",
    },
    waitForSelector: "div.results-list",
    screenshot: true,
  });

  return result;
});
```

### 2. Deep Site Crawling

Initiate recursive domain crawls with depth controls and path filtering:

```ts theme={null}
// app/routes/api.crawl.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { domain } = await request.json();

  const job = await client.crawl({
    url: domain,
    maxDepth: 3,
    maxPages: 100,
    allowBackwardLinks: false,
    allowExternalLinks: false,
    includePaths: ["/docs/**", "/guides/**"],
    excludePaths: ["/archive/**"],
    ignoreSitemap: false,
    webhook: {
      url: "https://my-remix-app.com/api/webhooks/spidra",
    },
  });

  return { jobId: job.jobId, status: "queued" };
});
```

### 3. Real-Time SERP Web Search

Query web, news, and image indexes to ground AI workflows with fresh search data:

```ts theme={null}
// app/routes/api.search.ts
import { createSpidraLoader } from "spidra-remix";

export const loader = createSpidraLoader(async ({ client, request }) => {
  const url = new URL(request.url);
  const query = url.searchParams.get("q") || "React Router v7 changes";

  const searchResults = await client.search({
    query,
    sources: ["web", "news"],
    limit: 10,
    filter: {
      domain: ["remix.run", "github.com"],
    },
  });

  return searchResults;
});
```

### 4. High-Throughput Batch Scraping

Scrape multiple independent pages in parallel:

```ts theme={null}
// app/routes/api.batch.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { urls } = await request.json();

  const batchJob = await client.batchScrape({
    urls,
    prompt: "Extract headline, author, and date published",
    output: "json",
    webhook: {
      url: "https://my-remix-app.com/api/webhooks/spidra",
    },
  });

  return { jobId: batchJob.jobId, status: "processing" };
});
```

### 5. Fast Domain Link Mapping

Discover all discoverable URLs across a domain without downloading full page bodies:

```ts theme={null}
// app/routes/api.map.ts
import { createSpidraLoader } from "spidra-remix";

export const loader = createSpidraLoader(async ({ client, request }) => {
  const url = new URL(request.url);
  const target = url.searchParams.get("url") || "https://remix.run";

  const siteMap = await client.map({
    url: target,
    limit: 500,
    search: "tutorial",
  });

  return siteMap;
});
```

### 6. Document Parsing

Extract clean markdown and tables from remote document URLs:

```ts theme={null}
// app/routes/api.parse.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { docUrl } = await request.json();

  const doc = await client.parse({
    url: docUrl,
    prompt: "Extract financial summary and executive commentary",
  });

  return { markdown: doc.markdown, metadata: doc.metadata };
});
```

### 7. Turnkey AI Templates & Scheduled Monitors

Execute pre-tuned extraction templates and automate recurring scrapes:

```ts theme={null}
// app/routes/api.templates.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  // Execute pre-tuned e-commerce template
  const productData = await client.executeTemplate("ecommerce-product", {
    url: "https://store.example.com/product/456",
  });

  // Schedule recurring monitor
  const monitor = await client.createMonitor({
    url: "https://status.remix.run",
    cron: "0 */6 * * *",
    prompt: "Check for any active incidents or maintenance notices",
    webhook: {
      url: "https://my-remix-app.com/api/webhooks/spidra",
    },
  });

  return { productData, monitorId: monitor.id };
});
```

## Typed Error Handling

`createSpidraAction()` and `createSpidraLoader()` automatically capture Spidra errors and return standardized JSON responses. You can also catch them manually:

```ts theme={null}
import {
  getSpidraClient,
  SpidraError,
  RateLimitError,
  TimeoutError,
  AuthenticationError,
  spidraJson,
} from "spidra-remix";

export const action = async () => {
  const client = getSpidraClient();

  try {
    const result = await client.scrape({
      urls: [{ url: "https://example.com" }],
      prompt: "Extract page title",
    });
    return spidraJson(result);
  } catch (error) {
    if (error instanceof RateLimitError) {
      return spidraJson(
        { error: "Rate limit exceeded", retryAfter: error.retryAfter },
        { status: 429 }
      );
    } else if (error instanceof AuthenticationError) {
      return spidraJson({ error: "Invalid API key" }, { status: 401 });
    } else if (error instanceof TimeoutError) {
      return spidraJson({ error: "Request timed out" }, { status: 504 });
    } else if (error instanceof SpidraError) {
      return spidraJson(
        { error: error.message },
        { status: error.status || 500 }
      );
    }
    return spidraJson({ error: "Internal server error" }, { status: 500 });
  }
};
```

## Related Frameworks

> [card] [Bun SDK](https://docs.spidra.io/sdks/bun.md)
>
> Native Bun scraping with Bun.file() parsing and Bun.serve handlers.

> [card] [Nuxt SDK](https://docs.spidra.io/sdks/nuxt.md)
>
> Nuxt 3 module and Nitro server route integration.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.