> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Rails

> Official Ruby on Rails integration for Spidra — AI web scraping, recursive crawling, SERP search, and document parsing with Railties, controller helpers, and signed webhook routing.

The Spidra Rails gem (`spidra-rails`) brings AI-powered web intelligence directly into Ruby on Rails:

* **Rails Railtie & Auto-Configuration**: Automatically binds configuration from `Rails.application.credentials` or environment variables.
* **`Spidra::Rails.client` & `Spidra.client`**: Global configured client instances ready for use across models, controllers, and background jobs.
* **Controller Concerns**: `Spidra::Rails::ControllerExtensions` for direct action helpers (`spidra_scrape`, `spidra_search`, `spidra_crawl`, etc.).
* **HMAC Webhook Routing**: One-line `spidra_webhooks at: "/spidra/webhooks"` route macro with constant-time HMAC-SHA256 signature verification.
* **ActiveSupport::Notifications**: Publishes `spidra.webhook_received` events for clean event-driven architecture.
* **Full API Surface**: Web Scraping, Recursive Crawling, SERP Search, Batch Scraping, Document Parsing, Domain Mapping, Turnkey Templates, Presets, and Monitors.

## Installation

Add this line to your application's `Gemfile`:

```ruby theme={null}
gem "spidra-rails"
```

Then execute:

```bash theme={null}
bundle install
```

> [!NOTE]
>
> Get your API key from [app.spidra.io](https://app.spidra.io) under **API Keys**.
> Store it in your Rails credentials or environment variables.

## Configuration

Create an initializer file at `config/initializers/spidra.rb`:

```ruby theme={null}
Spidra::Rails.configure do |config|
  # Your Spidra API Key (defaults to ENV["SPIDRA_API_KEY"] or credentials)
  config.api_key = ENV["SPIDRA_API_KEY"]

  # Webhook signing secret (defaults to ENV["SPIDRA_WEBHOOK_SECRET"] or credentials)
  config.webhook_secret = ENV["SPIDRA_WEBHOOK_SECRET"]

  # Optional overrides:
  config.base_url = ENV["SPIDRA_BASE_URL"] || "https://api.spidra.io/api"
  config.timeout  = (ENV["SPIDRA_TIMEOUT"] || 30).to_i
end
```

### Encrypted Credentials Support

Alternatively, you can store your credentials encrypted in `config/credentials.yml.enc`:

```yaml theme={null}
spidra:
  api_key: spd_live_your_api_key_here
  webhook_secret: whsec_your_signing_secret_here
```

`spidra-rails` automatically reads `Rails.application.credentials.spidra` when present.

## Quickstart

### 1. In a Controller

```ruby theme={null}
class ScrapesController < ApplicationController
  def create
    url = params.require(:url)

    result = Spidra::Rails.scrape(
      url,
      prompt: "Extract product title, price, in_stock status, and customer rating",
      output: "json"
    )

    render json: { product: result["content"] }
  end
end
```

### 2. In an ActiveJob Background Worker

```ruby theme={null}
class CrawlDocumentationJob < ApplicationJob
  queue_as :default

  def perform(docs_url)
    # Start crawl job asynchronously without blocking
    crawl = Spidra::Rails.crawl(
      url: docs_url,
      max_depth: 3,
      max_pages: 100,
      include_paths: ["/docs/**", "/guides/**"],
      webhook: {
        url: Rails.application.routes.url_helpers.spidra_webhooks_url
      }
    )

    Rails.logger.info("Crawl job queued with ID: #{crawl['jobId']}")
  end
end
```

### 3. In Rails Console / Runner

```ruby theme={null}
# Direct convenience helpers
results = Spidra::Rails.search("Ruby on Rails 8 release features", limit: 5)
puts results["results"]

# Check account credits
account = Spidra::Rails.account.get
puts "Credits remaining: #{account.dig('credits', 'remaining')}"
```

## Complete Spidra Features Guide

### 1. Scraping with Browser Actions & Proxies

Handle JavaScript-heavy SPAs, dynamic tables, and login screens using residential proxies:

```ruby theme={null}
result = Spidra::Rails.scrape(
  "https://example.com/interactive-pricing",
  prompt: "Extract plan names, monthly prices, and included feature list",
  output: "json",
  schema: {
    type: "object",
    properties: {
      plans: {
        type: "array",
        items: {
          type: "object",
          properties: {
            name: { type: "string" },
            price: { type: "number" },
            features: { type: "array", items: { type: "string" } }
          }
        }
      }
    }
  },
  actions: [
    { type: "click", selector: "button#annual-toggle" },
    { type: "wait", ms: 1500 },
    { type: "scroll", distance: 800 }
  ],
  proxy: {
    type: "residential",
    country: "US"
  },
  wait_for_selector: "div.pricing-tables",
  screenshot: true
)
```

### 2. Deep Site Crawling

Initiate recursive domain crawls with depth controls, path filters, and webhook callbacks:

```ruby theme={null}
# Start asynchronous crawl
crawl_job = Spidra::Rails.crawl(
  url: "https://rubyonrails.org",
  max_depth: 3,
  max_pages: 200,
  allow_backward_links: false,
  allow_external_links: false,
  include_paths: ["/docs/**"],
  exclude_paths: ["/docs/v5/**"],
  ignore_sitemap: false,
  webhook: {
    url: "https://my-rails-app.com/spidra/webhooks"
  }
)

job_id = crawl_job["jobId"]

# Check crawl status
status = Spidra::Rails.crawl.get(job_id)

# Fetch completed pages
pages = Spidra::Rails.crawl.pages(job_id, page: 1, limit: 50)
```

### 3. Real-Time SERP Web Search

Query web, news, and images to ground AI models with fresh web data:

```ruby theme={null}
search_data = Spidra::Rails.search(
  query: "Ruby on Rails performance tuning guide",
  sources: ["web", "news"],
  limit: 10,
  time_range: "month",
  country: "US",
  include_domains: ["rubyonrails.org", "github.com"]
)

search_data["results"].each do |item|
  puts "#{item['title']} - #{item['url']}"
end
```

### 4. High-Throughput Batch Scraping

Scrape multiple independent URLs concurrently in a single batch job:

```ruby theme={null}
batch_job = Spidra::Rails.batch(
  urls: [
    "https://news.ycombinator.com",
    "https://github.com/trending",
    "https://lobste.rs"
  ],
  prompt: "Extract top 5 trending posts with title, link, and score",
  output: "json",
  webhook: {
    url: "https://my-rails-app.com/spidra/webhooks"
  }
)

batch_id = batch_job["jobId"]
```

### 5. Fast Domain Link Mapping

Discover all discoverable URLs across a domain without downloading full page bodies:

```ruby theme={null}
site_map = Spidra::Rails.map(
  url: "https://rubyonrails.org",
  limit: 500,
  search: "doctrine"
)

puts "Discovered #{site_map['links'].length} URLs"
```

### 6. Document Parsing (PDF, Word, Excel, PPTX)

Extract clean markdown and tables from remote URLs or document files:

```ruby theme={null}
doc = Spidra::Rails.parse(
  url: "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
  mode: "ocr", # "auto" | "fast" | "ocr"
  prompt: "Extract executive summary and key numerical findings",
  include_pages: true
)

puts doc["markdown"]
```

### 7. Turnkey AI Templates & Scheduled Monitors

Execute pre-tuned extraction templates and schedule automated website monitors:

```ruby theme={null}
# 1. Run a pre-built template
product = Spidra::Rails.templates.run(
  "ecommerce-product",
  inputs: { url: "https://store.example.com/item/12345" }
)

# 2. Schedule recurring monitor
monitor = Spidra::Rails.monitors.create(
  url: "https://status.rubyonrails.org",
  cron: "0 */6 * * *",
  prompt: "Detect any service degradation or maintenance notices",
  webhook: {
    url: "https://my-rails-app.com/spidra/webhooks"
  }
)
```

## Webhook Routing & Notifications

Spidra sends asynchronous webhooks upon completion of crawl jobs, batch scrapes, or monitor alerts.

### 1. Register Webhook Route

In `config/routes.rb`:

```ruby theme={null}
Rails.application.routes.draw do
  # Registers POST /spidra/webhooks with automatic HMAC signature verification
  spidra_webhooks at: "/spidra/webhooks"
end
```

### 2. Subscribe to Webhook Events

`spidra-rails` automatically verifies the HMAC signature using `Spidra::Webhook.verify` and instruments an `ActiveSupport::Notifications` event.

Create an initializer or subscriber at `config/initializers/spidra_events.rb`:

```ruby theme={null}
ActiveSupport::Notifications.subscribe("spidra.webhook_received") do |_name, _start, _finish, _id, payload|
  data     = payload[:payload]
  event    = data["event"] || data["type"]
  job_id   = data["jobId"] || data.dig("data", "jobId")

  Rails.logger.info("Spidra Webhook [#{event}] received for job: #{job_id}")

  case event
  when "crawl.completed"
    ProcessCrawlResultsJob.perform_later(job_id, data)
  when "batch.completed"
    ProcessBatchResultsJob.perform_later(job_id, data)
  when "monitor.alert"
    NotifyTeamMonitorIncidentJob.perform_later(job_id, data)
  end
end
```

### 3. Custom Proc Handler (Optional)

You can also handle webhooks inline inside `config/initializers/spidra.rb`:

```ruby theme={null}
Spidra::Rails.configure do |config|
  config.on_webhook = lambda do |payload, request|
    Rails.logger.info "Inline webhook handler for #{payload['event']}"
  end
end
```

## Error Handling

All Spidra errors inherit from `Spidra::Error` and provide strongly-typed subclasses:

```ruby theme={null}
begin
  result = Spidra::Rails.scrape("https://example.com")
rescue Spidra::AuthenticationError => e
  Rails.logger.error("Spidra invalid API key: #{e.message}")
rescue Spidra::RateLimitError => e
  Rails.logger.warn("Spidra rate limit exceeded. Retry later.")
rescue Spidra::ForbiddenError => e
  Rails.logger.error("Spidra out of credits or forbidden.")
rescue Spidra::ServerError => e
  Rails.logger.error("Spidra server error: #{e.message}")
rescue Spidra::Error => e
  Rails.logger.error("Spidra API error [#{e.status}]: #{e.message}")
end
```

## Testing & Mocking in Rails

In your test suite, you can mock `Spidra::Rails.client` or individual methods:

### Minitest Example

```ruby theme={null}
require "test_helper"

class ScrapesControllerTest < ActionDispatch::IntegrationTest
  test "scrapes product successfully" do
    mock_result = { "content" => { "title" => "Mechanical Keyboard", "price" => 129.99 } }

    Spidra::Rails.client.scrape.stub :run, mock_result do
      post scrapes_url, params: { url: "https://example.com/keyboard" }
      assert_response :success
      assert_equal "Mechanical Keyboard", response.parsed_body.dig("product", "title")
    end
  end
end
```

## Related SDKs

> [card] [Ruby SDK](https://docs.spidra.io/sdks/ruby.md)
>
> Core Ruby SDK with zero dependencies for vanilla Ruby scripts.

> [card] [Laravel SDK](https://docs.spidra.io/sdks/laravel.md)
>
> Official Laravel package with Facades, Service Providers, and webhook events.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.