Skip to main content
The Spidra Rails gem (spidra-rails) brings AI-powered web intelligence directly into Ruby on Rails:
  • Rails Railtie & Auto-Configuration: Automatically binds configuration from Rails.application.credentials or environment variables.
  • Spidra::Rails.client & Spidra.client: Global configured client instances ready for use across models, controllers, and background jobs.
  • Controller Concerns: Spidra::Rails::ControllerExtensions for direct action helpers (spidra_scrape, spidra_search, spidra_crawl, etc.).
  • HMAC Webhook Routing: One-line spidra_webhooks at: "/spidra/webhooks" route macro with constant-time HMAC-SHA256 signature verification.
  • ActiveSupport::Notifications: Publishes spidra.webhook_received events for clean event-driven architecture.
  • Full API Surface: Web Scraping, Recursive Crawling, SERP Search, Batch Scraping, Document Parsing, Domain Mapping, Turnkey Templates, Presets, and Monitors.

Installation

Add this line to your application’s Gemfile:
Then execute:
Get your API key from app.spidra.io under API Keys. Store it in your Rails credentials or environment variables.

Configuration

Create an initializer file at config/initializers/spidra.rb:

Encrypted Credentials Support

Alternatively, you can store your credentials encrypted in config/credentials.yml.enc:
spidra-rails automatically reads Rails.application.credentials.spidra when present.

Quickstart

1. In a Controller

2. In an ActiveJob Background Worker

3. In Rails Console / Runner

Complete Spidra Features Guide

1. Scraping with Browser Actions & Proxies

Handle JavaScript-heavy SPAs, dynamic tables, and login screens using residential proxies:

2. Deep Site Crawling

Initiate recursive domain crawls with depth controls, path filters, and webhook callbacks:
Query web, news, and images to ground AI models with fresh web data:

4. High-Throughput Batch Scraping

Scrape multiple independent URLs concurrently in a single batch job:
Discover all discoverable URLs across a domain without downloading full page bodies:

6. Document Parsing (PDF, Word, Excel, PPTX)

Extract clean markdown and tables from remote URLs or document files:

7. Turnkey AI Templates & Scheduled Monitors

Execute pre-tuned extraction templates and schedule automated website monitors:

Webhook Routing & Notifications

Spidra sends asynchronous webhooks upon completion of crawl jobs, batch scrapes, or monitor alerts.

1. Register Webhook Route

In config/routes.rb:

2. Subscribe to Webhook Events

spidra-rails automatically verifies the HMAC signature using Spidra::Webhook.verify and instruments an ActiveSupport::Notifications event. Create an initializer or subscriber at config/initializers/spidra_events.rb:

3. Custom Proc Handler (Optional)

You can also handle webhooks inline inside config/initializers/spidra.rb:

Error Handling

All Spidra errors inherit from Spidra::Error and provide strongly-typed subclasses:

Testing & Mocking in Rails

In your test suite, you can mock Spidra::Rails.client or individual methods:

Minitest Example

Ruby SDK

Core Ruby SDK with zero dependencies for vanilla Ruby scripts.

Laravel SDK

Official Laravel package with Facades, Service Providers, and webhook events.