Browser action recipes

Every example on this page runs against a public site built for scraping practice: quotes.toscrape.com, books.toscrape.com, the-internet.herokuapp.com and scrapethissite.com. Send any of them to POST /scrape as they are, then poll GET /scrape/{jobId} for the result. For what each action does, see Browser actions.

1. No actions needed

When everything you want is on one page, you don’t need actions. Send the URL with a prompt.

{
  "urls": [{ "url": "https://quotes.toscrape.com" }],
  "prompt": "List every quote and its author. Return a JSON array: [{quote, author}]",
  "output": "json"
}

{
  "content": [
    { "quote": "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.", "author": "Albert Einstein" },
    { "quote": "It is our choices, Harry, that show what we truly are, far more than our abilities.", "author": "J.K. Rowling" },
    ...
  ]
}

2. Log in, then scrape

Fill in a login form, submit it, and wait for the logged-in page before Spidra reads it.

{
  "urls": [{
    "url": "https://quotes.toscrape.com/login",
    "actions": [
      { "type": "type",    "selector": "#username", "value": "demo" },
      { "type": "type",    "selector": "#password", "value": "demo" },
      { "type": "click",   "selector": "input[type='submit']" },
      { "type": "waitFor", "selector": "a[href='/logout']" }
    ]
  }],
  "prompt": "List the first 3 quotes with their authors",
  "output": "json"
}

The waitFor on the Logout link makes sure the login worked. If it didn’t, the job fails at step 4 and costs nothing. What you type is never sent to the AI and is masked in your logs. If you already have a session, cookies skip the form. Mark a step optional when it only applies sometimes, such as a cookie banner. If it can’t be done, it is skipped and the job goes on.

{
  "urls": [{
    "url": "https://quotes.toscrape.com",
    "actions": [
      { "type": "click", "selector": "#accept-cookies", "optional": true, "timeout": 2000 },
      { "type": "click", "selector": "li.next > a" }
    ]
  }]
}

This page has no banner, so the result says so:

{
  "warnings": ["Action 1 (click) was skipped: no element matches \"#accept-cookies\" (waited 2 seconds)."],
  "data": [{
    "actionResults": [
      { "index": 1, "type": "click", "status": "skipped", "durationMs": 2006, "error": "no element matches \"#accept-cookies\" (waited 2 seconds)." },
      { "index": 2, "type": "click", "status": "ok", "durationMs": 308 }
    ]
  }]
}

A short timeout keeps an optional step from waiting the full 10 seconds.

4. Search and press Enter

Type into a search box, press Enter, and wait for the results.

{
  "urls": [{
    "url": "https://www.scrapethissite.com/pages/forms/",
    "actions": [
      { "type": "type",    "selector": "#q", "value": "Boston" },
      { "type": "press",   "key": "Enter" },
      { "type": "waitFor", "selector": "tr.team" }
    ]
  }],
  "prompt": "List each team name, year and wins",
  "output": "json"
}

{
  "content": [
    { "team_name": "Boston Bruins", "year": 1990, "wins": 44 },
    { "team_name": "Boston Bruins", "year": 1991, "wins": 36 },
    { "team_name": "Boston Bruins", "year": 1992, "wins": 51 },
    ...
  ]
}

5. Wait for content that loads later

The button on this page starts a loader, and the text appears a few seconds later. waitFor moves on as soon as it shows up.

{
  "urls": [{
    "url": "https://the-internet.herokuapp.com/dynamic_loading/2",
    "actions": [
      { "type": "click",   "selector": "#start button" },
      { "type": "waitFor", "selector": "#finish", "timeout": 15000 }
    ]
  }]
}

The page content now includes Hello World!. Without the waitFor, Spidra would read the page before it loads.

6. Click with a plain-English description

When an element has no stable selector, describe it. The AI finds it, then Spidra clicks it.

{
  "urls": [{
    "url": "https://quotes.toscrape.com",
    "actions": [
      { "type": "click", "selector": "the Next page link at the bottom" }
    ]
  }]
}

This returns page 2. Finding the element uses a little AI, billed by tokens. select picks an option by its label or value. evaluate runs JavaScript and returns what it gives back, here the dropdown’s value.

{
  "urls": [{
    "url": "https://the-internet.herokuapp.com/dropdown",
    "actions": [
      { "type": "select",   "selector": "#dropdown", "value": "Option 2" },
      { "type": "evaluate", "script": "document.querySelector('#dropdown').value" }
    ]
  }]
}

"actionResults": [
  { "index": 1, "type": "select",   "status": "ok", "durationMs": 5 },
  { "index": 2, "type": "evaluate", "status": "ok", "durationMs": 1, "result": "2" }
]

8. Hover to reveal hidden text

{
  "urls": [{
    "url": "https://the-internet.herokuapp.com/hovers",
    "actions": [
      { "type": "hover",    "selector": ".figure" },
      { "type": "evaluate", "script": "document.querySelector('.figcaption h5').innerText" }
    ]
  }]
}

The evaluate step returns "name: user1", the caption that only shows while the mouse is over the first picture.

9. Load an infinite scroll page

scrollUntilEnd keeps scrolling until the page stops growing. This page loads 10 quotes at a time, up to 100.

{
  "urls": [{
    "url": "https://quotes.toscrape.com/scroll",
    "actions": [
      { "type": "scrollUntilEnd", "maxScrolls": 15, "pause": 1000 },
      { "type": "evaluate", "script": "document.querySelectorAll('.quote').length" }
    ]
  }]
}

"actionResults": [
  { "index": 1, "type": "scrollUntilEnd", "status": "ok", "durationMs": 11104, "result": { "scrolls": 11, "stoppedBecause": "end" } },
  { "index": 2, "type": "evaluate", "status": "ok", "durationMs": 2, "result": 100 }
]

On a page with a “Load more” button, add "clickSelector": "the Load more button" (or its CSS selector) and Spidra clicks it each round while it is there.

10. Return data with JavaScript

A script with statements needs return. The value comes back in actionResults, so you can collect exactly what you need without a prompt.

{
  "urls": [{
    "url": "https://quotes.toscrape.com",
    "actions": [
      { "type": "evaluate", "script": "const tags = [...document.querySelectorAll('.tag-item a')].map(a => a.innerText); return { count: tags.length, tags };" }
    ]
  }]
}

"result": { "count": 10, "tags": ["love", "inspirational", "life", "humor", "books", "reading", "friendship", "friends", "truth", "simile"] }

11. Screenshot partway through

{
  "urls": [{
    "url": "https://quotes.toscrape.com",
    "actions": [
      { "type": "click", "selector": "li.next > a" },
      { "type": "screenshot", "fullPage": true }
    ]
  }]
}

The screenshot step’s report has a screenshotUrl with an image of page 2.

12. forEach: read every item, across pages

inline mode reads each item where it is. A CSS captureSelector lists the items directly, and pagination follows the Next link.

{
  "urls": [{
    "url": "https://quotes.toscrape.com",
    "actions": [{
      "type": "forEach",
      "observe": "each quote block",
      "mode": "inline",
      "captureSelector": ".quote",
      "maxItems": 15,
      "pagination": { "nextSelector": "li.next > a", "maxPages": 2 }
    }]
  }]
}

## Item 1

“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
by Albert Einstein (about)
Tags: change deep-thoughts thinking world

---

## Item 2

“It is our choices, Harry, that show what we truly are, far more than our abilities.”
by J.K. Rowling (about)
Tags: abilities choices

---

...

That is 10 quotes from page 1 and 5 from page 2. maxPages counts the first page.

13. forEach: open each item’s page

navigate mode opens the page each item links to. itemPrompt pulls fields out of each page as it is read.

{
  "urls": [{
    "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html",
    "actions": [{
      "type": "forEach",
      "observe": "each book title link in the product grid",
      "mode": "navigate",
      "captureSelector": "article.product_page",
      "maxItems": 3,
      "waitAfterClick": 500,
      "itemPrompt": "Return JSON: {title, price, availability}"
    }]
  }]
}

## Item 1

{
  "title": "It's Only the Himalayas",
  "price": "£45.17",
  "availability": "In stock (19 available)"
}

---

## Item 2

{
  "title": "Full Moon over Noah’s Ark: An Odyssey to Mount Ararat and Beyond",
  "price": "£49.43",
  "availability": "In stock (15 available)"
}

---

## Item 3

{
  "title": "See America: A Celebration of Our National Parks & Treasured Sites",
  "price": "£48.87",
  "availability": "In stock (14 available)"
}

Each page opened costs 1 credit on top of the 1 for the URL, and itemPrompt adds its tokens.

14. Click to a category first, then forEach

Actions before a forEach run once. Here a plain-English click opens the Science category, and the forEach reads its books.

{
  "urls": [{
    "url": "https://books.toscrape.com",
    "actions": [
      { "type": "click", "selector": "the Science category in the left sidebar" },
      {
        "type": "forEach",
        "observe": "each book card",
        "mode": "inline",
        "captureSelector": "article.product_pod",
        "maxItems": 5,
        "itemPrompt": "Return JSON: {title, price}"
      }
    ]
  }]
}

15. forEach: click each item and read what loads

click mode clicks each item and reads what it opens on the same page. On this page, clicking a year loads that year’s films into a table.

{
  "urls": [{
    "url": "https://www.scrapethissite.com/pages/ajax-javascript/",
    "actions": [{
      "type": "forEach",
      "observe": "each year link (2015, 2014, 2013 ...)",
      "mode": "click",
      "captureSelector": "table.table",
      "maxItems": 2,
      "waitAfterClick": 2000
    }]
  }]
}

Item 1 is the 2015 table (Spotlight, Mad Max: Fury Road, The Revenant and the rest) and item 2 is 2014.

16. forEach: run steps on each item

actions inside a forEach run on each item after it is opened. Here each book page is scrolled to the bottom before it is read.

{
  "urls": [{
    "url": "https://books.toscrape.com/catalogue/category/books/poetry_23/index.html",
    "actions": [{
      "type": "forEach",
      "observe": "each book title link in the product grid",
      "mode": "navigate",
      "captureSelector": "article.product_page",
      "maxItems": 2,
      "waitAfterClick": 500,
      "actions": [
        { "type": "scroll", "to": "bottom" }
      ],
      "itemPrompt": "Return JSON: {title, price, description}"
    }]
  }]
}

17. Several categories in one request

A request takes up to 3 URLs, each with its own actions. They run in parallel and data has one entry per URL.

{
  "urls": [
    {
      "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
      "actions": [{ "type": "forEach", "observe": "each book card", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 3 }]
    },
    {
      "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html",
      "actions": [{ "type": "forEach", "observe": "each book card", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 3 }]
    },
    {
      "url": "https://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
      "actions": [{ "type": "forEach", "observe": "each book card", "mode": "inline", "captureSelector": "article.product_pod", "maxItems": 3 }]
    }
  ],
  "prompt": "Return a JSON array of every book: {category, title, price}",
  "output": "json"
}

18. forEach plus a final prompt

itemPrompt works on one item at a time. A top-level prompt then runs once over all of them, here to sort the list.

{
  "urls": [{
    "url": "https://quotes.toscrape.com",
    "actions": [{
      "type": "forEach",
      "observe": "each quote block",
      "mode": "inline",
      "captureSelector": ".quote",
      "maxItems": 20,
      "pagination": { "nextSelector": "li.next > a", "maxPages": 2 }
    }]
  }],
  "prompt": "Return a JSON array of {author, quote}, sorted by author name",
  "output": "json"
}

Choosing a pattern

You want to	Use
Read a list that fits on one page	A top-level `prompt`, no actions (1)
Get past a login form	`type`, `click`, then `waitFor` (2)
Dismiss something that is only sometimes there	`optional: true` (3)
Submit a search	`type` and `press` `Enter` (4)
Wait for content that loads late	`waitFor` (5)
Load an infinite scroll or “Load more” page	`scrollUntilEnd` (9)
Read a value the page holds	`evaluate` (7, 10)
Collect items across results pages	forEach `inline` with `pagination` (12, 18)
Get details from each item’s own page	forEach `navigate` (13, 16)
Read content that opens on a click	forEach `click` (15)

Get started

Tools

Ready-made data

Getting the data you want

Supported sites

Guides

Dashboard

Account and billing

Browser action recipes

1. No actions needed

2. Log in, then scrape

3. Click away a banner only if it’s there

4. Search and press Enter

5. Wait for content that loads later

6. Click with a plain-English description

8. Hover to reveal hidden text

9. Load an infinite scroll page

10. Return data with JavaScript

11. Screenshot partway through

12. forEach: read every item, across pages

13. forEach: open each item’s page

14. Click to a category first, then forEach

15. forEach: click each item and read what loads

16. forEach: run steps on each item

17. Several categories in one request

18. forEach plus a final prompt

Choosing a pattern

​1. No actions needed

​2. Log in, then scrape

​3. Click away a banner only if it’s there

​4. Search and press Enter

​5. Wait for content that loads later

​6. Click with a plain-English description

​7. Pick a dropdown option and read it back

​8. Hover to reveal hidden text

​9. Load an infinite scroll page

​10. Return data with JavaScript

​11. Screenshot partway through

​12. forEach: read every item, across pages

​13. forEach: open each item’s page

​14. Click to a category first, then forEach

​15. forEach: click each item and read what loads

​16. forEach: run steps on each item

​17. Several categories in one request

​18. forEach plus a final prompt

​Choosing a pattern

1. No actions needed

2. Log in, then scrape

3. Click away a banner only if it’s there

4. Search and press Enter

5. Wait for content that loads later

6. Click with a plain-English description

7. Pick a dropdown option and read it back

8. Hover to reveal hidden text

9. Load an infinite scroll page

10. Return data with JavaScript

11. Screenshot partway through

12. forEach: read every item, across pages

13. forEach: open each item’s page

14. Click to a category first, then forEach

15. forEach: click each item and read what loads

16. forEach: run steps on each item

17. Several categories in one request

18. forEach plus a final prompt

Choosing a pattern