Remix

The Spidra Remix integration (spidra-remix) brings web data intelligence to fullstack Remix and React Router v7 applications:

Loaders & Actions: createSpidraLoader() and createSpidraAction() generators with automatic typed error responses.
HMAC Webhook Handler: handleRemixWebhook() validates asynchronous job events and dispatches typed callbacks.
Remix Response Helper: spidraJson() produces standard Remix Response objects with proper HTTP status codes.
Full API Surface: Scrape, Crawl, SERP Search, Batch Scraping, Document Parsing, Domain Mapping, Turnkey Templates, Presets, and Monitors.

Installation

npm install spidra-remix spidra

Get your API key from app.spidra.io under API Keys. Store it in .env or set SPIDRA_API_KEY.

Configuration

Set environment variables in .env:

SPIDRA_API_KEY=spd_live_your_api_key
SPIDRA_WEBHOOK_SECRET=whsec_your_signing_secret

# Optional overrides
SPIDRA_BASE_URL=https://api.spidra.io/api
SPIDRA_MAX_RETRIES=3
SPIDRA_BACKOFF_FACTOR=1.0

getSpidraClient() automatically reads SPIDRA_API_KEY from process.env.

Quickstart

1. In a Remix Action

Handle form submissions and extract structured data:

// app/routes/scrape.tsx
import type { ActionFunctionArgs } from "@remix-run/node";
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const formData = await request.formData();
  const url = formData.get("url") as string;

  return await client.scrape({
    urls: [{ url }],
    prompt: "Extract product title, price, and customer review rating",
    output: "json",
  });
});

2. In a Remix Loader

Pre-fetch search or crawl data on the server before rendering:

// app/routes/search.tsx
import type { LoaderFunctionArgs } from "@remix-run/node";
import { createSpidraLoader } from "spidra-remix";

export const loader = createSpidraLoader(async ({ client, request }) => {
  const url = new URL(request.url);
  const q = url.searchParams.get("q") || "Remix fullstack development";

  return await client.search({
    query: q,
    sources: ["web", "news"],
    limit: 10,
  });
});

3. Asynchronous Webhooks

Handle asynchronous crawl or batch job completions securely:

// app/routes/api.webhooks.spidra.ts
import type { ActionFunctionArgs } from "@remix-run/node";
import { handleRemixWebhook } from "spidra-remix";

export const action = async ({ request }: ActionFunctionArgs) => {
  return await handleRemixWebhook({
    request,
    webhookSecret: process.env.SPIDRA_WEBHOOK_SECRET,
    onEvent: async (event) => {
      console.log(`Received verified webhook event: ${event.event}`);

      if (event.event === "crawl.completed") {
        console.log(`Crawl job ${event.jobId} completed!`);
      }
    },
  });
};

Complete Spidra Features Guide

1. Scraping with Browser Actions & Proxies

Bypass anti-bot mechanisms with residential proxies and interact with dynamic pages before extraction:

// app/routes/api.scrape-dynamic.ts
import type { ActionFunctionArgs } from "@remix-run/node";
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { targetUrl } = await request.json();

  const result = await client.scrape({
    urls: [{ url: targetUrl }],
    prompt: "Extract interactive table items and prices",
    output: "json",
    schema: {
      type: "object",
      properties: {
        items: {
          type: "array",
          items: {
            type: "object",
            properties: {
              title: { type: "string" },
              price: { type: "number" },
            },
          },
        },
      },
    },
    actions: [
      { type: "click", selector: "button#load-more" },
      { type: "wait", ms: 2000 },
      { type: "scroll", distance: 1000 },
    ],
    proxy: {
      type: "residential",
      country: "US",
    },
    waitForSelector: "div.results-list",
    screenshot: true,
  });

  return result;
});

2. Deep Site Crawling

Initiate recursive domain crawls with depth controls and path filtering:

// app/routes/api.crawl.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { domain } = await request.json();

  const job = await client.crawl({
    url: domain,
    maxDepth: 3,
    maxPages: 100,
    allowBackwardLinks: false,
    allowExternalLinks: false,
    includePaths: ["/docs/**", "/guides/**"],
    excludePaths: ["/archive/**"],
    ignoreSitemap: false,
    webhook: {
      url: "https://my-remix-app.com/api/webhooks/spidra",
    },
  });

  return { jobId: job.jobId, status: "queued" };
});

3. Real-Time SERP Web Search

Query web, news, and image indexes to ground AI workflows with fresh search data:

// app/routes/api.search.ts
import { createSpidraLoader } from "spidra-remix";

export const loader = createSpidraLoader(async ({ client, request }) => {
  const url = new URL(request.url);
  const query = url.searchParams.get("q") || "React Router v7 changes";

  const searchResults = await client.search({
    query,
    sources: ["web", "news"],
    limit: 10,
    filter: {
      domain: ["remix.run", "github.com"],
    },
  });

  return searchResults;
});

4. High-Throughput Batch Scraping

Scrape multiple independent pages in parallel:

// app/routes/api.batch.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { urls } = await request.json();

  const batchJob = await client.batchScrape({
    urls,
    prompt: "Extract headline, author, and date published",
    output: "json",
    webhook: {
      url: "https://my-remix-app.com/api/webhooks/spidra",
    },
  });

  return { jobId: batchJob.jobId, status: "processing" };
});

5. Fast Domain Link Mapping

Discover all discoverable URLs across a domain without downloading full page bodies:

// app/routes/api.map.ts
import { createSpidraLoader } from "spidra-remix";

export const loader = createSpidraLoader(async ({ client, request }) => {
  const url = new URL(request.url);
  const target = url.searchParams.get("url") || "https://remix.run";

  const siteMap = await client.map({
    url: target,
    limit: 500,
    search: "tutorial",
  });

  return siteMap;
});

6. Document Parsing

Extract clean markdown and tables from remote document URLs:

// app/routes/api.parse.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  const { docUrl } = await request.json();

  const doc = await client.parse({
    url: docUrl,
    prompt: "Extract financial summary and executive commentary",
  });

  return { markdown: doc.markdown, metadata: doc.metadata };
});

7. Turnkey AI Templates & Scheduled Monitors

Execute pre-tuned extraction templates and automate recurring scrapes:

// app/routes/api.templates.ts
import { createSpidraAction } from "spidra-remix";

export const action = createSpidraAction(async ({ client, request }) => {
  // Execute pre-tuned e-commerce template
  const productData = await client.executeTemplate("ecommerce-product", {
    url: "https://store.example.com/product/456",
  });

  // Schedule recurring monitor
  const monitor = await client.createMonitor({
    url: "https://status.remix.run",
    cron: "0 */6 * * *",
    prompt: "Check for any active incidents or maintenance notices",
    webhook: {
      url: "https://my-remix-app.com/api/webhooks/spidra",
    },
  });

  return { productData, monitorId: monitor.id };
});

Typed Error Handling

createSpidraAction() and createSpidraLoader() automatically capture Spidra errors and return standardized JSON responses. You can also catch them manually:

import {
  getSpidraClient,
  SpidraError,
  RateLimitError,
  TimeoutError,
  AuthenticationError,
  spidraJson,
} from "spidra-remix";

export const action = async () => {
  const client = getSpidraClient();

  try {
    const result = await client.scrape({
      urls: [{ url: "https://example.com" }],
      prompt: "Extract page title",
    });
    return spidraJson(result);
  } catch (error) {
    if (error instanceof RateLimitError) {
      return spidraJson(
        { error: "Rate limit exceeded", retryAfter: error.retryAfter },
        { status: 429 }
      );
    } else if (error instanceof AuthenticationError) {
      return spidraJson({ error: "Invalid API key" }, { status: 401 });
    } else if (error instanceof TimeoutError) {
      return spidraJson({ error: "Request timed out" }, { status: 504 });
    } else if (error instanceof SpidraError) {
      return spidraJson(
        { error: error.message },
        { status: error.status || 500 }
      );
    }
    return spidraJson({ error: "Internal server error" }, { status: 500 });
  }
};

Bun SDK

Native Bun scraping with Bun.file() parsing and Bun.serve handlers.

Nuxt SDK

Nuxt 3 module and Nitro server route integration.

SDKs

AI tools

Installation

Configuration

Quickstart

1. In a Remix Action

2. In a Remix Loader

3. Asynchronous Webhooks

Complete Spidra Features Guide

1. Scraping with Browser Actions & Proxies

2. Deep Site Crawling

3. Real-Time SERP Web Search

4. High-Throughput Batch Scraping

5. Fast Domain Link Mapping

6. Document Parsing

7. Turnkey AI Templates & Scheduled Monitors

Typed Error Handling

Bun SDK

Nuxt SDK

Installation

Configuration

Quickstart

1. In a Remix Action

2. In a Remix Loader

3. Asynchronous Webhooks

Complete Spidra Features Guide

1. Scraping with Browser Actions & Proxies

2. Deep Site Crawling

3. Real-Time SERP Web Search

4. High-Throughput Batch Scraping

5. Fast Domain Link Mapping

6. Document Parsing

7. Turnkey AI Templates & Scheduled Monitors

Typed Error Handling

Related Frameworks

Bun SDK

Nuxt SDK