Scrape pages behind a login

​

Pass your session cookies

1. Sign in to the site in Chrome, Edge or Firefox.
2. Open DevTools (F12), go to Application (Chrome, Edge) or Storage (Firefox), then Cookies, and pick the site.
3. Copy the cookies: either the ones that look like a session (session, auth, token, sid), or select every row and copy them all.
4. Send them in cookies:

{
  "urls": [{ "url": "https://app.example.com/reports" }],
  "prompt": "Extract the monthly traffic summary",
  "cookies": "session_id=abc123xyz; auth_token=eyJhbGciOiJ..."
}

​

Log in with browser actions

{
  "urls": [{
    "url": "https://quotes.toscrape.com/login",
    "actions": [
      { "type": "type", "selector": "#username", "value": "demo" },
      { "type": "type", "selector": "#password", "value": "demo" },
      { "type": "click", "selector": "input[type='submit']" },
      { "type": "waitFor", "selector": "a[href='/logout']" }
    ]
  }],
  "prompt": "List the first 3 quotes with their authors",
  "output": "json"
}

​

Good to know

• Cookies are never stored. They’re used for the job and dropped. That’s also why a batch retry needs them sent again.
• Sessions expire. If pages start coming back logged out, copy fresh cookies.
• Copy from the right domain. Cookies belong to one site; copy them from the domain you’re scraping.
• Use your own accounts. You’re responsible for following each site’s terms and the law, including for content behind a login.