Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.spidra.io/llms.txt

Use this file to discover all available pages before exploring further.

Some websites require you to be logged in to see certain content. Authenticated scraping lets you access this content by providing your session cookies to Spidra.

​
When You Need It

Use authenticated scraping when:
  • The content is behind a login page
  • You see different data when logged in vs logged out
  • The website requires a subscription or account
  • You want to scrape your own account data

​
How It Works

  1. Log into the target website in your browser
  2. Copy your session cookies
  3. Include them in your API request
  4. Spidra uses those cookies to access the page as if it were you

​
Getting Your Cookies

​
Step 1: Log In

Open the website in Chrome, Firefox, or Edge and log into your account.

​
Step 2: Open DevTools

Press F12 or right-click → Inspect → go to the Application tab (Chrome) or Storage tab (Firefox).

​
Step 3: Find Cookies

Click Cookies in the left sidebar, then select the website domain.

​
Step 4: Copy Cookies

You have two options: Option A - Copy specific cookies: Look for authentication cookies (often named session, auth, token, or similar). Copy the name and value of each. Option B - Copy all cookies: Select all rows (Ctrl/Cmd+A), then copy (Ctrl/Cmd+C).
Spidra accepts cookies in two formats. The API auto-detects which format you’re using.

​
Standard Format

The traditional name=value format, separated by semicolons: 
{
  "urls": [{"url": "https://app.example.com/dashboard"}],
  "prompt": "Extract my account details",
  "output": "json",
  "cookies": "session_id=abc123xyz; auth_token=eyJhbGciOiJ..."
}

​
Raw DevTools Format

Paste directly from Chrome DevTools without any formatting: 
{
  "urls": [{"url": "https://app.example.com/dashboard"}],
  "prompt": "Extract my account details",
  "output": "json",
  "cookies": "session_id\tabc123xyz\t.example.com\t/\t2026-12-31\t50\t✓\t✓\tLax\nauth_token\teyJhbGciOiJ...\t.example.com\t/\t2026-12-31\t200\t✓\t✓\tStrict"
}
The API extracts the cookie name, value, domain, and path from each row. Extra columns like Size, HttpOnly, and SameSite are ignored.

​
Examples

​
Scraping a dashboard

{
  "urls": [{"url": "https://analytics.example.com/reports"}],
  "prompt": "Extract the monthly traffic summary",
  "output": "json",
  "cookies": "session=eyJ...; csrf_token=abc123"
}

​
Crawling authenticated pages

{
  "baseUrl": "https://app.example.com/projects",
  "crawlInstruction": "Find all project pages",
  "transformInstruction": "Extract project name, status, and last updated date",
  "maxPages": 20,
  "cookies": "auth=eyJ...; user_id=12345"
}

​
Tips

  • Session expiry: Cookies expire. If your scrape fails with authentication errors, get fresh cookies.
  • Required cookies: You usually don’t need all cookies. Look for ones with names like session, auth, token, jwt, or user.
  • Domain matching: Cookies are domain-specific. Make sure you’re copying cookies from the correct domain.
  • Incognito test: Before scraping, try opening the URL in an incognito window with your cookies to verify they work.
You are responsible for ensuring your authenticated scraping complies with applicable laws and the website’s Terms of Service. Only scrape content you’re authorized to access. Cookies are processed transiently and never stored by Spidra.

Submit a Scrape Job

API reference

Submit a Crawl Job

Crawl with authentication