> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Browser Actions
> Automate clicks, scrolls, form fills, and element loops before scraping.
## What Are Browser Actions?
When Spidra opens a page, it does not just grab the raw HTML and leave. You can tell it to interact with the page first. That means clicking buttons, filling in search boxes, scrolling down to load more content, dismissing cookie banners, or looping through every card, accordion, or link on the page.
Actions run in the order you provide them, one after the other. Spidra uses a real browser, so anything a human could do on the page, your action pipeline can do too.
You add actions in the `actions` array on each URL object:
```json theme={null}
{
"urls": [{
"url": "https://example.com/products",
"actions": [
{ "type": "click", "selector": "#accept-cookies" },
{ "type": "wait", "duration": 1500 },
{ "type": "scroll", "to": "80%" }
]
}],
"prompt": "List all products and their prices"
}
```
***
## How to Target Elements
Most actions need to know which element on the page to interact with. There are two ways to point Spidra at an element.
### CSS or XPath selectors (the `selector` field)
If you know the page structure, CSS selectors are the most precise and reliable way to target an element.
```json theme={null}
{ "type": "click", "selector": "#accept-cookies" }
{ "type": "click", "selector": ".load-more-btn" }
{ "type": "click", "selector": "button[data-testid='submit']" }
{ "type": "type", "selector": "input[name='q']", "value": "laptop" }
```
XPath works too:
```json theme={null}
{ "type": "click", "selector": "//button[contains(text(), 'Accept')]" }
```
### Plain English descriptions (the `value` field)
You can also describe the element in plain English and Spidra will find it on the page for you. This is useful when CSS selectors are fragile or change between page loads.
```json theme={null}
{ "type": "click", "value": "Accept cookies button" }
{ "type": "click", "value": "Load more products button at the bottom of the page" }
{ "type": "check", "value": "Subscribe to newsletter checkbox" }
```
For `click`, `check`, and `uncheck` you can use either `selector` (CSS or XPath) or `value` (plain English description). For `type`, use `selector` to point at the input field and `value` for the text you want to type.
***
## Actions Reference
### click
Clicks any element on the page. This works for buttons, links, tabs, dropdowns, toggles, and anything else that responds to a click.
Either `selector` or `value` is required.
```json theme={null}
{ "type": "click", "selector": "#accept-cookies" }
{ "type": "click", "value": "Close the cookie consent modal" }
{ "type": "click", "selector": "//button[text()='Load More']" }
```
Common uses:
* Dismiss cookie consent banners before the real content loads
* Click "Load More" to reveal additional results
* Open dropdown menus or select tabs
* Navigate to the next page
***
### type
Types text into an input field, textarea, or search box.
Both `selector` and `value` are required.
```json theme={null}
{ "type": "type", "selector": "input[name='q']", "value": "best laptops 2024" }
{ "type": "type", "selector": "#email", "value": "test@example.com" }
{ "type": "type", "selector": "input[placeholder='Search...']", "value": "running shoes" }
```
A common pattern is to type a query, click submit, then wait for results:
```json theme={null}
{
"actions": [
{ "type": "type", "selector": "#search-input", "value": "running shoes" },
{ "type": "click", "selector": "button[type='submit']" },
{ "type": "wait", "duration": 2000 }
]
}
```
***
### check
Checks a checkbox. If the checkbox is already checked, nothing happens.
```json theme={null}
{ "type": "check", "selector": "#agree-terms" }
{ "type": "check", "value": "In Stock filter checkbox" }
```
***
### uncheck
Unchecks a checkbox. If the checkbox is already unchecked, nothing happens.
```json theme={null}
{ "type": "uncheck", "selector": "#newsletter" }
{ "type": "uncheck", "value": "Show out-of-stock items checkbox" }
```
***
### wait
Pauses the scrape for a set number of milliseconds. Use this after actions that trigger loading, animations, or data fetching.
The `duration` field sets the wait time in milliseconds.
```json theme={null}
{ "type": "wait", "duration": 2000 }
{ "type": "wait", "duration": 500 }
```
Add a wait after clicking "Load More" or scrolling to the bottom. This gives the browser time to fetch and render the new content before Spidra captures the page.
The older `value` field (e.g. `"value": 2000`) is still accepted but deprecated. Use `duration` going forward.
***
### scroll
Scrolls the page to a percentage of its total height. This is essential for pages that load content as you scroll (infinite scroll, lazy-loaded images).
The `to` field takes a number or string percentage between 0 and 100.
```json theme={null}
{ "type": "scroll", "to": "50%" }
{ "type": "scroll", "to": 80 }
{ "type": "scroll", "to": "100%" }
```
A pattern for pages with lazy-loaded content:
```json theme={null}
{
"actions": [
{ "type": "scroll", "to": "50%" },
{ "type": "wait", "duration": 1500 },
{ "type": "scroll", "to": "100%" },
{ "type": "wait", "duration": 1500 }
]
}
```
The older `value` field (e.g. `"value": "80%"`) is still accepted but deprecated. Use `to` going forward.
***
## forEach: Process Every Element on a Page
`forEach` is the most powerful action in Spidra. Rather than scraping a page once and hoping all the data is there, `forEach` finds a set of matching elements on the page and processes each one individually. It then combines all the results into one output.
Think of it as running a mini scrape on every item in a list.
### Adding forEach
`forEach` is an action, so it goes inside the `actions` array along with any other actions you want to run first. Use the `observe` field to describe which elements to find.
```json theme={null}
{
"urls": [{
"url": "https://example.com/products",
"actions": [
{ "type": "click", "selector": "#filter-electronics" },
{
"type": "forEach",
"observe": "Find all product cards",
"mode": "inline",
"maxItems": 20,
"itemPrompt": "Extract product name and price. Return as JSON: {name, price}"
}
]
}]
}
```
Any actions listed before the `forEach` run once on the page first. The `forEach` then runs on whatever state the page is in after those actions complete.
### Writing a good observe instruction
The `observe` field is the most important part of forEach. It tells Spidra which elements to find on the page. Vague instructions produce inconsistent results.
**Describe what the elements are, not what you want to do with them:**
```
Good: "Find all book cards in the product grid"
Good: "Find all quote blocks on the page"
Good: "Find all room type cards"
Too vague: "Find items"
Action-oriented (avoid): "Click every product"
```
**Be specific about location and type:**
```
Good: "Find all article elements with class product_pod"
Good: "Find all list items in the search results container"
Good: "Find all anchor tags inside the product grid"
```
**In inline mode with a CSS `captureSelector`:** the `observe` instruction helps Spidra locate the set of elements to iterate over, but the `captureSelector` CSS is what actually reads each element's content. Keep them consistent. If `observe` says "product cards" then `captureSelector` should point at the same element (e.g. `article.product_pod`), not a child element.
***
### Do you actually need forEach?
Before reaching for forEach, consider whether a top-level `prompt` is enough.
**If the list is short and fits on one page, you usually do not need forEach at all.** Just scrape the URL and use `prompt` to extract what you need from the full page. It is simpler and works just as well.
```json theme={null}
{
"urls": [{ "url": "https://quotes.toscrape.com" }],
"prompt": "List all quotes and their authors"
}
```
**Use forEach when:**
* You need to collect items across multiple pages using `pagination`. A top-level prompt only sees the page it lands on. It cannot follow next-page links on its own.
* You have 20 or more items and want `itemPrompt` to extract fields from each one individually, keeping each AI call small and the output consistently structured.
* You are using `navigate` or `click` mode to access content that is only available after clicking into each item.
***
### The three forEach modes
The `mode` field tells Spidra how to interact with each element it finds.
#### inline mode: Read the element directly
Use this when the data is visible on the page inside each matched element, and you do not need to click anything. Product cards, quote blocks, search result rows, table rows.
Spidra reads each element's content and moves on. The page is not changed between items.
```json theme={null}
{
"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
"actions": [{
"type": "forEach",
"observe": "Find all book cards",
"mode": "inline",
"captureSelector": "article.product_pod",
"maxItems": 10,
"itemPrompt": "Extract title, price, and star rating. Return as JSON: {title, price, star_rating}"
}]
}
```
Sample response:
```
## Item 1
{"title": "Sharp Objects", "price": "£47.82", "star_rating": "Four"}
---
## Item 2
{"title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One"}
---
## Item 3
{"title": "When We Collided", "price": "£31.77", "star_rating": "One"}
```
For a short single-page list, `inline` mode without `pagination` or `itemPrompt` gives you structured output, but a top-level `prompt` on the raw page produces equivalent results with less setup. The main reasons to use inline are pagination across pages and per-item AI extraction at scale (20+ items).
***
#### navigate mode: Follow each link to its destination page
Use this when each element is a link and the content you want lives on the page it points to. Product listings, search results, category pages where clicking a card takes you to the full detail page.
Spidra clicks each element, loads the destination page, captures it, then returns and moves to the next element.
```json theme={null}
{
"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
"actions": [{
"type": "forEach",
"observe": "Find all book title links in the product grid",
"mode": "navigate",
"captureSelector": "article.product_page",
"maxItems": 6,
"waitAfterClick": 800,
"itemPrompt": "Extract title, price, star rating (One through Five), and availability. Return as JSON."
}]
}
```
Sample response:
```
## Item 1
{
"title": "Sharp Objects",
"price": "£47.82",
"star_rating": "Four",
"availability": "In stock"
}
---
## Item 2
{
"title": "In a Dark, Dark Wood",
"price": "£19.63",
"star_rating": "One",
"availability": "In stock"
}
```
Navigate mode is slower than inline because it loads a full page per item. But it gives you access to all the rich detail that only exists on the individual item page, like full descriptions, specifications, reviews, and related content.
***
#### click mode: Click to expand, capture, then move on
Use this for pages where clicking an element opens more content within the same page. Hotel room cards that open modals, FAQ rows that expand, product variant selectors that reveal details.
Spidra clicks each element, waits for the expanded content to appear, captures it, then closes and moves to the next.
```json theme={null}
{
"url": "https://hotels.example.com/hotel/grand-plaza",
"actions": [{
"type": "forEach",
"observe": "Find all room category cards",
"mode": "click",
"captureSelector": "[role='dialog']",
"waitAfterClick": 1200,
"itemPrompt": "Extract room name, bed type, price per night, and amenities. Return as JSON."
}]
}
```
Sample response:
```
## Item 1
{
"room": "Deluxe King Room",
"bed_type": "1 King Bed",
"price_per_night": "$189",
"amenities": ["Free WiFi", "City view", "Air conditioning", "Mini bar"]
}
---
## Item 2
{
"room": "Standard Twin Room",
"bed_type": "2 Twin Beds",
"price_per_night": "$129",
"amenities": ["Free WiFi", "Garden view", "Air conditioning"]
}
```
If no `captureSelector` is provided, Spidra first tries to find an open modal (`[role="dialog"]`) and falls back to capturing the full page.
***
### captureSelector
This tells Spidra which part of the page to read for each item. Without it, Spidra captures whatever is most relevant by default, but being specific gives you cleaner, more focused results.
You can use a CSS selector or a plain English description:
```json theme={null}
"captureSelector": "article.product_pod"
"captureSelector": "article.product_page"
"captureSelector": "[role='dialog']"
"captureSelector": "The pricing table in the center of the page"
```
If you leave it out:
* In `click` mode: Spidra looks for an open modal first, then captures the full page
* In `navigate` mode: Spidra captures the full destination page
* In `inline` mode: Spidra captures the full HTML of each matched element
CSS selectors are more reliable for `captureSelector`. When you use plain English, Spidra tries to locate the element using AI, but if it cannot find a match it falls back to capturing the full page content instead. For consistent production results, use a CSS selector when you know the structure of the page.
***
### itemPrompt: Extract specific fields from each item
`itemPrompt` runs an AI extraction on each item individually, right after it is captured. You tell it exactly what fields to pull out and in what format.
This is different from the top-level `prompt`, which runs once on the full combined output after all items are collected.
Why use itemPrompt:
* Each item is processed on its own, so the AI has full focus on just that one item
* Very useful in navigate mode where each destination page has a lot of unrelated content
* Keeps output clean and structured per item from the start
* The top-level prompt still runs afterwards if you also provide one
```json theme={null}
"itemPrompt": "Extract the book title and price. Return as JSON: {title, price}"
"itemPrompt": "Extract title, price in £XX.XX format, star rating as a word (One through Five), and availability. Return as JSON."
"itemPrompt": "Get the room name, nightly rate, and list of included amenities. Return as JSON."
```
Without `itemPrompt`, each item's raw captured content is returned as markdown text, and your top-level `prompt` handles all the extraction at the end.
***
### pagination: Keep going to the next page
After processing all elements on the current page, forEach can follow the next-page link and continue collecting until it hits your limit or runs out of pages.
You need to tell it which button or link goes to the next page:
```json theme={null}
{
"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
"actions": [{
"type": "forEach",
"observe": "Find all book title links",
"mode": "navigate",
"maxItems": 20,
"itemPrompt": "Extract title, price, and rating as JSON",
"pagination": {
"nextSelector": "li.next > a",
"maxPages": 3
}
}]
}
```
How it works: once all items on the first page are collected, Spidra clicks the next page button, waits for the new items to load, and continues. It stops when it has collected `maxItems` total across all pages, when it has visited `maxPages` additional pages, or when there is no next page button left.
`nextSelector` examples:
```json theme={null}
"nextSelector": "li.next > a"
"nextSelector": "a[rel='next']"
"nextSelector": ".pagination .next"
"nextSelector": "The Next page button at the bottom right"
```
`maxPages` is the number of extra pages beyond the first one. Setting `maxPages: 3` means Spidra processes the starting page plus 3 more, so 4 pages total.
***
### Per-element actions: Do something on each item before capturing
The `actions` field inside forEach lets you run browser actions after landing on each item but before capturing its content. This is useful when the destination page needs a scroll to load the full content, an extra click to expand a section, or a moment to settle before reading.
```json theme={null}
{
"url": "https://books.toscrape.com",
"actions": [
{ "type": "click", "selector": "a[href='catalogue/category/books/poetry_23/index.html']" },
{
"type": "forEach",
"observe": "Find all book title links in the product grid",
"mode": "navigate",
"captureSelector": "article.product_page",
"maxItems": 3,
"waitAfterClick": 1000,
"actions": [
{ "type": "scroll", "to": "50%" }
],
"itemPrompt": "Extract title, price, star rating, and the full product description. Return as JSON: {title, price, star_rating, description}"
}
]
}
```
Sample response:
```
## Item 1
{
"title": "Poetry Unbound: 50 Poems to Open Your World",
"price": "£23.00",
"star_rating": "Five",
"description": "Selected and introduced by Padraig O Tuama, this anthology brings together..."
}
```
***
### maxItems and waitAfterClick
**`maxItems`** sets a cap on how many elements to process. The default is 50 and the maximum allowed is also 50. This limit applies across all pages combined when you use pagination.
```json theme={null}
"maxItems": 10
"maxItems": 50
```
**`waitAfterClick`** is how long to wait in milliseconds after clicking or navigating before capturing the content. The default is 2500ms. You can lower this for fast static pages or raise it for pages that fetch content from an API after load.
```json theme={null}
"waitAfterClick": 500 // Static pages
"waitAfterClick": 1200 // Pages with animations
"waitAfterClick": 3000 // Slow or API-driven content
```
***
## Advanced Patterns
### Pattern 1: Click to a category first, then forEach over its items
The pre-actions run once when the URL loads. The forEach runs on whatever page the browser is on after those actions finish.
```json theme={null}
{
"urls": [{
"url": "https://books.toscrape.com",
"actions": [
{ "type": "click", "selector": "a[href='catalogue/category/books/travel_2/index.html']" },
{
"type": "forEach",
"observe": "Find all book title links in the product grid",
"mode": "navigate",
"captureSelector": "article.product_page",
"maxItems": 4,
"waitAfterClick": 800,
"itemPrompt": "Extract the book title and price. Return as JSON: {title, price}"
}
]
}],
"output": "json"
}
```
Result, 4 Travel books with title and price:
```json theme={null}
{
"data": [
{
"url": "https://books.toscrape.com",
"success": true,
"markdownContent": "## Item 1\n\n{\"title\": \"It's Only the Himalayas\", \"price\": \"£45.17\"}\n\n---\n\n## Item 2\n\n{\"title\": \"Full Moon over Noah's Ark\", \"price\": \"£49.43\"}\n\n---\n\n## Item 3\n\n..."
}
]
}
```
***
### Pattern 2: Click to a category using plain English, then forEach with pagination
When you do not know the CSS selector for a navigation element, describe it in plain English using the `value` field on a `click` action. This is the same `click` action covered in the Actions Reference — no special setup needed.
```json theme={null}
{
"urls": [{
"url": "https://books.toscrape.com",
"actions": [
{ "type": "click", "value": "Science category in the left sidebar" },
{
"type": "forEach",
"observe": "Find all product cards",
"mode": "inline",
"captureSelector": "article.product_pod",
"maxItems": 12,
"itemPrompt": "Extract book title and price. Return as JSON: {title, price}",
"pagination": {
"nextSelector": "li.next > a",
"maxPages": 1
}
}
]
}]
}
```
***
### Pattern 3: Scrape multiple categories at the same time
Pass up to 3 URL objects in a single request and they are all processed in parallel. Each URL has its own forEach config.
```json theme={null}
{
"urls": [
{
"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
"actions": [{
"type": "forEach",
"observe": "Find all book cards",
"mode": "inline",
"captureSelector": "article.product_pod",
"maxItems": 4,
"itemPrompt": "Return JSON: {title, price, category: 'Mystery'}"
}]
},
{
"url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html",
"actions": [{
"type": "forEach",
"observe": "Find all book cards",
"mode": "inline",
"captureSelector": "article.product_pod",
"maxItems": 4,
"itemPrompt": "Return JSON: {title, price, category: 'Travel'}"
}]
},
{
"url": "https://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
"actions": [{
"type": "forEach",
"observe": "Find all book cards",
"mode": "inline",
"captureSelector": "article.product_pod",
"maxItems": 4,
"itemPrompt": "Return JSON: {title, price, category: 'Historical Fiction'}"
}]
}
]
}
```
The response `data` array has three entries, one per URL, each with their 4 extracted books.
Maximum 3 URLs per request. Each URL counts as 1 credit.
***
### Pattern 4: Click to category, navigate each book, scroll to load full description
Pre-action clicks into a category. forEach navigates into each book's detail page. A per-element scroll reveals the full description. The AI extracts everything.
```json theme={null}
{
"urls": [{
"url": "https://books.toscrape.com",
"actions": [
{ "type": "click", "selector": "a[href='catalogue/category/books/poetry_23/index.html']" },
{
"type": "forEach",
"observe": "Find all book title links in the product grid",
"mode": "navigate",
"captureSelector": "article.product_page",
"maxItems": 3,
"waitAfterClick": 1000,
"actions": [
{ "type": "scroll", "to": "50%" }
],
"itemPrompt": "Extract the book title, price, star rating (One through Five), and the full product description paragraph. Return as JSON: {title, price, star_rating, description}"
}
]
}]
}
```
What happens step by step:
1. Opens the homepage in a real browser
2. Clicks the Poetry category link
3. Finds all book title links on the category page
4. For each book (up to 3): opens the book page, waits 1 second, scrolls down 50% to reveal the full description, captures the product section, runs AI to extract the four fields
5. Combines all results into a single output with numbered items
***
## Response Format
All forEach results are returned in the `markdownContent` field of each URL's result. Items are numbered from 1 and separated by `---`.
Without `itemPrompt`, you get the raw captured content per item:
```
## Item 1
# Sharp Objects
Price: £47.82
Rating: Four stars
Stock: In stock
A gripping psychological thriller...
---
## Item 2
# In a Dark, Dark Wood
Price: £19.63
...
```
With `itemPrompt`, each item has already been extracted by the AI before being combined:
```
## Item 1
{"title": "Sharp Objects", "price": "£47.82", "star_rating": "Four", "availability": "In stock"}
---
## Item 2
{"title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One", "availability": "In stock"}
```
If you also provide a top-level `prompt` with `output: "json"`, the AI reads the combined output and returns a clean final JSON array in the `content` field of the response.
***
## itemPrompt vs Top-level prompt
Both are optional but they serve different purposes and work well together.
| | `itemPrompt` | Top-level `prompt` |
| -------------------- | ------------------------------------------------------ | ---------------------------------------------------------- |
| When it runs | Right after each item is captured, during scraping | Once, after all items are collected and combined |
| What it sees | Only that one item's content | All items together |
| Best for | Per-item field extraction, cleaning up noisy pages | Final restructuring, filtering, or summarising all results |
| Where output appears | Each item's section in `result.data[].markdownContent` | `result.content` in the response |
They do not conflict. They run at completely different stages. `itemPrompt` runs during scraping, one item at a time. The top-level `prompt` runs after all scraping is finished, on the full combined output. If you use both, `itemPrompt` cleans up each item first, then the top-level prompt does a final pass on all the cleaned results together.
***
## Limits
| Setting | Default | Maximum |
| --------------------- | ------- | ------- |
| `maxItems` | 50 | 50 |
| `pagination.maxPages` | 5 | 10 |
| URLs per request | 1 | 3 |
***
## Full forEach Field Reference
```json theme={null}
{
"actions": [{
"type": "forEach",
"observe": "Describe the elements to find, e.g. Find all product cards",
"mode": "inline",
"captureSelector": "article.product_pod",
"maxItems": 20,
"waitAfterClick": 1000,
"actions": [
{ "type": "scroll", "to": "50%" }
],
"itemPrompt": "Extract X, Y, Z. Return as JSON: {x, y, z}",
"pagination": {
"nextSelector": "li.next > a",
"maxPages": 3
}
}]
}
```
| Field | Type | Required | Description |
| ------------------------- | ------ | ----------------------- | ------------------------------------------------------------------------------------------------ |
| `observe` | string | Yes | Plain English description of the elements to find and process |
| `mode` | string | No | How to interact with each element. One of `inline`, `navigate`, or `click`. Defaults to `click`. |
| `captureSelector` | string | No | CSS selector or plain English description of which part of the page to capture per item |
| `maxItems` | number | No | Maximum number of elements to process. Default is 50, maximum is 50. |
| `waitAfterClick` | number | No | Milliseconds to wait after clicking or navigating before capturing. Default is 2500. |
| `actions` | array | No | Browser actions to run on each item after clicking or navigating, before capturing |
| `itemPrompt` | string | No | AI extraction prompt to run on each item individually |
| `pagination.nextSelector` | string | Yes if using pagination | CSS selector or plain English description of the next page button |
| `pagination.maxPages` | number | No | How many additional pages to process beyond the first. Default is 5, maximum is 10. |