spidra-rails) brings AI-powered web intelligence directly into Ruby on Rails:
- Rails Railtie & Auto-Configuration: Automatically binds configuration from
Rails.application.credentialsor environment variables. Spidra::Rails.client&Spidra.client: Global configured client instances ready for use across models, controllers, and background jobs.- Controller Concerns:
Spidra::Rails::ControllerExtensionsfor direct action helpers (spidra_scrape,spidra_search,spidra_crawl, etc.). - HMAC Webhook Routing: One-line
spidra_webhooks at: "/spidra/webhooks"route macro with constant-time HMAC-SHA256 signature verification. - ActiveSupport::Notifications: Publishes
spidra.webhook_receivedevents for clean event-driven architecture. - Full API Surface: Web Scraping, Recursive Crawling, SERP Search, Batch Scraping, Document Parsing, Domain Mapping, Turnkey Templates, Presets, and Monitors.
Installation
Add this line to your application’sGemfile:
Get your API key from app.spidra.io under API Keys.
Store it in your Rails credentials or environment variables.
Configuration
Create an initializer file atconfig/initializers/spidra.rb:
Encrypted Credentials Support
Alternatively, you can store your credentials encrypted inconfig/credentials.yml.enc:
spidra-rails automatically reads Rails.application.credentials.spidra when present.
Quickstart
1. In a Controller
2. In an ActiveJob Background Worker
3. In Rails Console / Runner
Complete Spidra Features Guide
1. Scraping with Browser Actions & Proxies
Handle JavaScript-heavy SPAs, dynamic tables, and login screens using residential proxies:2. Deep Site Crawling
Initiate recursive domain crawls with depth controls, path filters, and webhook callbacks:3. Real-Time SERP Web Search
Query web, news, and images to ground AI models with fresh web data:4. High-Throughput Batch Scraping
Scrape multiple independent URLs concurrently in a single batch job:5. Fast Domain Link Mapping
Discover all discoverable URLs across a domain without downloading full page bodies:6. Document Parsing (PDF, Word, Excel, PPTX)
Extract clean markdown and tables from remote URLs or document files:7. Turnkey AI Templates & Scheduled Monitors
Execute pre-tuned extraction templates and schedule automated website monitors:Webhook Routing & Notifications
Spidra sends asynchronous webhooks upon completion of crawl jobs, batch scrapes, or monitor alerts.1. Register Webhook Route
Inconfig/routes.rb:
2. Subscribe to Webhook Events
spidra-rails automatically verifies the HMAC signature using Spidra::Webhook.verify and instruments an ActiveSupport::Notifications event.
Create an initializer or subscriber at config/initializers/spidra_events.rb:
3. Custom Proc Handler (Optional)
You can also handle webhooks inline insideconfig/initializers/spidra.rb:
Error Handling
All Spidra errors inherit fromSpidra::Error and provide strongly-typed subclasses:
Testing & Mocking in Rails
In your test suite, you can mockSpidra::Rails.client or individual methods:
Minitest Example
Related SDKs
Ruby SDK
Core Ruby SDK with zero dependencies for vanilla Ruby scripts.
Laravel SDK
Official Laravel package with Facades, Service Providers, and webhook events.

