Browser Actions

What Are Browser Actions?

When Spidra opens a page, it does not just grab the raw HTML and leave. You can tell it to interact with the page first. That means clicking buttons, filling in search boxes, scrolling down to load more content, dismissing cookie banners, or looping through every card, accordion, or link on the page. Actions run in the order you provide them, one after the other. Spidra uses a real browser, so anything a human could do on the page, your action pipeline can do too. You add actions in the actions array on each URL object:

{
  "urls": [{
    "url": "https://example.com/products",
    "actions": [
      { "type": "click",  "selector": "#accept-cookies" },
      { "type": "wait",   "duration": 1500 },
      { "type": "scroll", "to": "80%" }
    ]
  }],
  "prompt": "List all products and their prices"
}

How to Target Elements

Most actions need to know which element on the page to interact with. There are two ways to point Spidra at an element.

CSS or XPath selectors (the `selector` field)

If you know the page structure, CSS selectors are the most precise and reliable way to target an element.

{ "type": "click", "selector": "#accept-cookies" }
{ "type": "click", "selector": ".load-more-btn" }
{ "type": "click", "selector": "button[data-testid='submit']" }
{ "type": "type",  "selector": "input[name='q']", "value": "laptop" }

XPath works too:

{ "type": "click", "selector": "//button[contains(text(), 'Accept')]" }

Plain English descriptions (the `value` field)

You can also describe the element in plain English and Spidra will find it on the page for you. This is useful when CSS selectors are fragile or change between page loads.

{ "type": "click",  "value": "Accept cookies button" }
{ "type": "click",  "value": "Load more products button at the bottom of the page" }
{ "type": "check",  "value": "Subscribe to newsletter checkbox" }

For click, check, and uncheck you can use either selector (CSS or XPath) or value (plain English description). For type, use selector to point at the input field and value for the text you want to type.

Actions Reference

click

Clicks any element on the page. This works for buttons, links, tabs, dropdowns, toggles, and anything else that responds to a click. Either selector or value is required.

{ "type": "click", "selector": "#accept-cookies" }

{ "type": "click", "value": "Close the cookie consent modal" }

{ "type": "click", "selector": "//button[text()='Load More']" }

Common uses:

Dismiss cookie consent banners before the real content loads
Click “Load More” to reveal additional results
Open dropdown menus or select tabs
Navigate to the next page

type

Types text into an input field, textarea, or search box. Both selector and value are required.

{ "type": "type", "selector": "input[name='q']", "value": "best laptops 2024" }

{ "type": "type", "selector": "#email", "value": "[email protected]" }

{ "type": "type", "selector": "input[placeholder='Search...']", "value": "running shoes" }

A common pattern is to type a query, click submit, then wait for results:

{
  "actions": [
    { "type": "type",  "selector": "#search-input", "value": "running shoes" },
    { "type": "click", "selector": "button[type='submit']" },
    { "type": "wait",  "duration": 2000 }
  ]
}

check

Checks a checkbox. If the checkbox is already checked, nothing happens.

{ "type": "check", "selector": "#agree-terms" }
{ "type": "check", "value": "In Stock filter checkbox" }

uncheck

Unchecks a checkbox. If the checkbox is already unchecked, nothing happens.

{ "type": "uncheck", "selector": "#newsletter" }
{ "type": "uncheck", "value": "Show out-of-stock items checkbox" }

wait

Pauses the scrape for a set number of milliseconds. Use this after actions that trigger loading, animations, or data fetching. The duration field sets the wait time in milliseconds.

{ "type": "wait", "duration": 2000 }

{ "type": "wait", "duration": 500 }

Add a wait after clicking “Load More” or scrolling to the bottom. This gives the browser time to fetch and render the new content before Spidra captures the page.

The older value field (e.g. "value": 2000) is still accepted but deprecated. Use duration going forward.

scroll

Scrolls the page to a percentage of its total height. This is essential for pages that load content as you scroll (infinite scroll, lazy-loaded images). The to field takes a number or string percentage between 0 and 100.

{ "type": "scroll", "to": "50%" }

{ "type": "scroll", "to": 80 }

{ "type": "scroll", "to": "100%" }

A pattern for pages with lazy-loaded content:

{
  "actions": [
    { "type": "scroll", "to": "50%" },
    { "type": "wait",   "duration": 1500 },
    { "type": "scroll", "to": "100%" },
    { "type": "wait",   "duration": 1500 }
  ]
}

The older value field (e.g. "value": "80%") is still accepted but deprecated. Use to going forward.

forEach: Process Every Element on a Page

forEach is the most powerful action in Spidra. Rather than scraping a page once and hoping all the data is there, forEach finds a set of matching elements on the page and processes each one individually. It then combines all the results into one output. Think of it as running a mini scrape on every item in a list.

Adding forEach

forEach is an action, so it goes inside the actions array along with any other actions you want to run first. Use the observe field to describe which elements to find.

{
  "urls": [{
    "url": "https://example.com/products",
    "actions": [
      { "type": "click", "selector": "#filter-electronics" },
      {
        "type":       "forEach",
        "observe":    "Find all product cards",
        "mode":       "inline",
        "maxItems":   20,
        "itemPrompt": "Extract product name and price. Return as JSON: {name, price}"
      }
    ]
  }]
}

Any actions listed before the forEach run once on the page first. The forEach then runs on whatever state the page is in after those actions complete.

Writing a good observe instruction

The observe field is the most important part of forEach. It tells Spidra which elements to find on the page. Vague instructions produce inconsistent results. Describe what the elements are, not what you want to do with them:

Good: "Find all book cards in the product grid"
Good: "Find all quote blocks on the page"
Good: "Find all room type cards"

Too vague: "Find items"
Action-oriented (avoid): "Click every product"

Be specific about location and type:

Good: "Find all article elements with class product_pod"
Good: "Find all list items in the search results container"
Good: "Find all anchor tags inside the product grid"

In inline mode with a CSS captureSelector: the observe instruction helps Spidra locate the set of elements to iterate over, but the captureSelector CSS is what actually reads each element’s content. Keep them consistent. If observe says “product cards” then captureSelector should point at the same element (e.g. article.product_pod), not a child element.

Do you actually need forEach?

Before reaching for forEach, consider whether a top-level prompt is enough. If the list is short and fits on one page, you usually do not need forEach at all. Just scrape the URL and use prompt to extract what you need from the full page. It is simpler and works just as well.

{
  "urls": [{ "url": "https://quotes.toscrape.com" }],
  "prompt": "List all quotes and their authors"
}

Use forEach when:

You need to collect items across multiple pages using pagination. A top-level prompt only sees the page it lands on. It cannot follow next-page links on its own.
You have 20 or more items and want itemPrompt to extract fields from each one individually, keeping each AI call small and the output consistently structured.
You are using navigate or click mode to access content that is only available after clicking into each item.

The three forEach modes

The mode field tells Spidra how to interact with each element it finds.

inline mode: Read the element directly

Use this when the data is visible on the page inside each matched element, and you do not need to click anything. Product cards, quote blocks, search result rows, table rows. Spidra reads each element’s content and moves on. The page is not changed between items.

{
  "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
  "actions": [{
    "type":            "forEach",
    "observe":         "Find all book cards",
    "mode":            "inline",
    "captureSelector": "article.product_pod",
    "maxItems":        10,
    "itemPrompt":      "Extract title, price, and star rating. Return as JSON: {title, price, star_rating}"
  }]
}

Sample response:

## Item 1

{"title": "Sharp Objects", "price": "£47.82", "star_rating": "Four"}

---

## Item 2

{"title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One"}

---

## Item 3

{"title": "When We Collided", "price": "£31.77", "star_rating": "One"}

For a short single-page list, inline mode without pagination or itemPrompt gives you structured output, but a top-level prompt on the raw page produces equivalent results with less setup. The main reasons to use inline are pagination across pages and per-item AI extraction at scale (20+ items).

navigate mode: Follow each link to its destination page

Use this when each element is a link and the content you want lives on the page it points to. Product listings, search results, category pages where clicking a card takes you to the full detail page. Spidra clicks each element, loads the destination page, captures it, then returns and moves to the next element.

{
  "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
  "actions": [{
    "type":            "forEach",
    "observe":         "Find all book title links in the product grid",
    "mode":            "navigate",
    "captureSelector": "article.product_page",
    "maxItems":        6,
    "waitAfterClick":  800,
    "itemPrompt":      "Extract title, price, star rating (One through Five), and availability. Return as JSON."
  }]
}

Sample response:

## Item 1

{
  "title": "Sharp Objects",
  "price": "£47.82",
  "star_rating": "Four",
  "availability": "In stock"
}

---

## Item 2

{
  "title": "In a Dark, Dark Wood",
  "price": "£19.63",
  "star_rating": "One",
  "availability": "In stock"
}

Navigate mode is slower than inline because it loads a full page per item. But it gives you access to all the rich detail that only exists on the individual item page, like full descriptions, specifications, reviews, and related content.

click mode: Click to expand, capture, then move on

Use this for pages where clicking an element opens more content within the same page. Hotel room cards that open modals, FAQ rows that expand, product variant selectors that reveal details. Spidra clicks each element, waits for the expanded content to appear, captures it, then closes and moves to the next.

{
  "url": "https://hotels.example.com/hotel/grand-plaza",
  "actions": [{
    "type":            "forEach",
    "observe":         "Find all room category cards",
    "mode":            "click",
    "captureSelector": "[role='dialog']",
    "waitAfterClick":  1200,
    "itemPrompt":      "Extract room name, bed type, price per night, and amenities. Return as JSON."
  }]
}

Sample response:

## Item 1

{
  "room": "Deluxe King Room",
  "bed_type": "1 King Bed",
  "price_per_night": "$189",
  "amenities": ["Free WiFi", "City view", "Air conditioning", "Mini bar"]
}

---

## Item 2

{
  "room": "Standard Twin Room",
  "bed_type": "2 Twin Beds",
  "price_per_night": "$129",
  "amenities": ["Free WiFi", "Garden view", "Air conditioning"]
}

If no captureSelector is provided, Spidra first tries to find an open modal ([role="dialog"]) and falls back to capturing the full page.

captureSelector

This tells Spidra which part of the page to read for each item. Without it, Spidra captures whatever is most relevant by default, but being specific gives you cleaner, more focused results. You can use a CSS selector or a plain English description:

"captureSelector": "article.product_pod"
"captureSelector": "article.product_page"
"captureSelector": "[role='dialog']"
"captureSelector": "The pricing table in the center of the page"

If you leave it out:

In click mode: Spidra looks for an open modal first, then captures the full page
In navigate mode: Spidra captures the full destination page
In inline mode: Spidra captures the full HTML of each matched element

CSS selectors are more reliable for captureSelector. When you use plain English, Spidra tries to locate the element using AI, but if it cannot find a match it falls back to capturing the full page content instead. For consistent production results, use a CSS selector when you know the structure of the page.

itemPrompt: Extract specific fields from each item

itemPrompt runs an AI extraction on each item individually, right after it is captured. You tell it exactly what fields to pull out and in what format. This is different from the top-level prompt, which runs once on the full combined output after all items are collected. Why use itemPrompt:

Each item is processed on its own, so the AI has full focus on just that one item
Very useful in navigate mode where each destination page has a lot of unrelated content
Keeps output clean and structured per item from the start
The top-level prompt still runs afterwards if you also provide one

"itemPrompt": "Extract the book title and price. Return as JSON: {title, price}"
"itemPrompt": "Extract title, price in £XX.XX format, star rating as a word (One through Five), and availability. Return as JSON."
"itemPrompt": "Get the room name, nightly rate, and list of included amenities. Return as JSON."

Without itemPrompt, each item’s raw captured content is returned as markdown text, and your top-level prompt handles all the extraction at the end.

pagination: Keep going to the next page

After processing all elements on the current page, forEach can follow the next-page link and continue collecting until it hits your limit or runs out of pages. You need to tell it which button or link goes to the next page:

{
  "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
  "actions": [{
    "type":       "forEach",
    "observe":    "Find all book title links",
    "mode":       "navigate",
    "maxItems":   20,
    "itemPrompt": "Extract title, price, and rating as JSON",
    "pagination": {
      "nextSelector": "li.next > a",
      "maxPages":     3
    }
  }]
}

How it works: once all items on the first page are collected, Spidra clicks the next page button, waits for the new items to load, and continues. It stops when it has collected maxItems total across all pages, when it has visited maxPages additional pages, or when there is no next page button left. nextSelector examples:

"nextSelector": "li.next > a"
"nextSelector": "a[rel='next']"
"nextSelector": ".pagination .next"
"nextSelector": "The Next page button at the bottom right"

maxPages is the number of extra pages beyond the first one. Setting maxPages: 3 means Spidra processes the starting page plus 3 more, so 4 pages total.

Per-element actions: Do something on each item before capturing

The actions field inside forEach lets you run browser actions after landing on each item but before capturing its content. This is useful when the destination page needs a scroll to load the full content, an extra click to expand a section, or a moment to settle before reading.

{
  "url": "https://books.toscrape.com",
  "actions": [
    { "type": "click", "selector": "a[href='catalogue/category/books/poetry_23/index.html']" },
    {
      "type":            "forEach",
      "observe":         "Find all book title links in the product grid",
      "mode":            "navigate",
      "captureSelector": "article.product_page",
      "maxItems":        3,
      "waitAfterClick":  1000,
      "actions": [
        { "type": "scroll", "to": "50%" }
      ],
      "itemPrompt": "Extract title, price, star rating, and the full product description. Return as JSON: {title, price, star_rating, description}"
    }
  ]
}

Sample response:

## Item 1

{
  "title": "Poetry Unbound: 50 Poems to Open Your World",
  "price": "£23.00",
  "star_rating": "Five",
  "description": "Selected and introduced by Padraig O Tuama, this anthology brings together..."
}

maxItems and waitAfterClick

maxItems sets a cap on how many elements to process. The default is 50 and the maximum allowed is also 50. This limit applies across all pages combined when you use pagination.

"maxItems": 10
"maxItems": 50

waitAfterClick is how long to wait in milliseconds after clicking or navigating before capturing the content. The default is 2500ms. You can lower this for fast static pages or raise it for pages that fetch content from an API after load.

"waitAfterClick": 500    // Static pages
"waitAfterClick": 1200   // Pages with animations
"waitAfterClick": 3000   // Slow or API-driven content

Advanced Patterns

Pattern 1: Click to a category first, then forEach over its items

The pre-actions run once when the URL loads. The forEach runs on whatever page the browser is on after those actions finish.

{
  "urls": [{
    "url": "https://books.toscrape.com",
    "actions": [
      { "type": "click", "selector": "a[href='catalogue/category/books/travel_2/index.html']" },
      {
        "type":            "forEach",
        "observe":         "Find all book title links in the product grid",
        "mode":            "navigate",
        "captureSelector": "article.product_page",
        "maxItems":        4,
        "waitAfterClick":  800,
        "itemPrompt":      "Extract the book title and price. Return as JSON: {title, price}"
      }
    ]
  }],
  "output": "json"
}

Result, 4 Travel books with title and price:

{
  "data": [
    {
      "url": "https://books.toscrape.com",
      "success": true,
      "markdownContent": "## Item 1\n\n{\"title\": \"It's Only the Himalayas\", \"price\": \"£45.17\"}\n\n---\n\n## Item 2\n\n{\"title\": \"Full Moon over Noah's Ark\", \"price\": \"£49.43\"}\n\n---\n\n## Item 3\n\n..."
    }
  ]
}

Pattern 2: Click to a category using plain English, then forEach with pagination

When you do not know the CSS selector for a navigation element, describe it in plain English using the value field on a click action. This is the same click action covered in the Actions Reference — no special setup needed.

{
  "urls": [{
    "url": "https://books.toscrape.com",
    "actions": [
      { "type": "click", "value": "Science category in the left sidebar" },
      {
        "type":            "forEach",
        "observe":         "Find all product cards",
        "mode":            "inline",
        "captureSelector": "article.product_pod",
        "maxItems":        12,
        "itemPrompt":      "Extract book title and price. Return as JSON: {title, price}",
        "pagination": {
          "nextSelector": "li.next > a",
          "maxPages":     1
        }
      }
    ]
  }]
}

Pattern 3: Scrape multiple categories at the same time

Pass up to 3 URL objects in a single request and they are all processed in parallel. Each URL has its own forEach config.

{
  "urls": [
    {
      "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
      "actions": [{
        "type":            "forEach",
        "observe":         "Find all book cards",
        "mode":            "inline",
        "captureSelector": "article.product_pod",
        "maxItems":        4,
        "itemPrompt":      "Return JSON: {title, price, category: 'Mystery'}"
      }]
    },
    {
      "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html",
      "actions": [{
        "type":            "forEach",
        "observe":         "Find all book cards",
        "mode":            "inline",
        "captureSelector": "article.product_pod",
        "maxItems":        4,
        "itemPrompt":      "Return JSON: {title, price, category: 'Travel'}"
      }]
    },
    {
      "url": "https://books.toscrape.com/catalogue/category/books/historical-fiction_4/index.html",
      "actions": [{
        "type":            "forEach",
        "observe":         "Find all book cards",
        "mode":            "inline",
        "captureSelector": "article.product_pod",
        "maxItems":        4,
        "itemPrompt":      "Return JSON: {title, price, category: 'Historical Fiction'}"
      }]
    }
  ]
}

The response data array has three entries, one per URL, each with their 4 extracted books.

Maximum 3 URLs per request. Each URL counts as 1 credit.

Pattern 4: Click to category, navigate each book, scroll to load full description

Pre-action clicks into a category. forEach navigates into each book’s detail page. A per-element scroll reveals the full description. The AI extracts everything.

{
  "urls": [{
    "url": "https://books.toscrape.com",
    "actions": [
      { "type": "click", "selector": "a[href='catalogue/category/books/poetry_23/index.html']" },
      {
        "type":            "forEach",
        "observe":         "Find all book title links in the product grid",
        "mode":            "navigate",
        "captureSelector": "article.product_page",
        "maxItems":        3,
        "waitAfterClick":  1000,
        "actions": [
          { "type": "scroll", "to": "50%" }
        ],
        "itemPrompt": "Extract the book title, price, star rating (One through Five), and the full product description paragraph. Return as JSON: {title, price, star_rating, description}"
      }
    ]
  }]
}

What happens step by step:

Opens the homepage in a real browser
Clicks the Poetry category link
Finds all book title links on the category page
For each book (up to 3): opens the book page, waits 1 second, scrolls down 50% to reveal the full description, captures the product section, runs AI to extract the four fields
Combines all results into a single output with numbered items

Response Format

All forEach results are returned in the markdownContent field of each URL’s result. Items are numbered from 1 and separated by ---. Without itemPrompt, you get the raw captured content per item:

## Item 1

# Sharp Objects

Price: £47.82
Rating: Four stars
Stock: In stock

A gripping psychological thriller...

---

## Item 2

# In a Dark, Dark Wood

Price: £19.63
...

With itemPrompt, each item has already been extracted by the AI before being combined:

## Item 1

{"title": "Sharp Objects", "price": "£47.82", "star_rating": "Four", "availability": "In stock"}

---

## Item 2

{"title": "In a Dark, Dark Wood", "price": "£19.63", "star_rating": "One", "availability": "In stock"}

If you also provide a top-level prompt with output: "json", the AI reads the combined output and returns a clean final JSON array in the content field of the response.

itemPrompt vs Top-level prompt

Both are optional but they serve different purposes and work well together.

	`itemPrompt`	Top-level `prompt`
When it runs	Right after each item is captured, during scraping	Once, after all items are collected and combined
What it sees	Only that one item’s content	All items together
Best for	Per-item field extraction, cleaning up noisy pages	Final restructuring, filtering, or summarising all results
Where output appears	Each item’s section in `result.data[].markdownContent`	`result.content` in the response

They do not conflict. They run at completely different stages. itemPrompt runs during scraping, one item at a time. The top-level prompt runs after all scraping is finished, on the full combined output. If you use both, itemPrompt cleans up each item first, then the top-level prompt does a final pass on all the cleaned results together.

Limits

Setting	Default	Maximum
`maxItems`	50	50
`pagination.maxPages`	5	10
URLs per request	1	3

Full forEach Field Reference

{
  "actions": [{
    "type":            "forEach",
    "observe":         "Describe the elements to find, e.g. Find all product cards",
    "mode":            "inline",
    "captureSelector": "article.product_pod",
    "maxItems":        20,
    "waitAfterClick":  1000,
    "actions": [
      { "type": "scroll", "to": "50%" }
    ],
    "itemPrompt":      "Extract X, Y, Z. Return as JSON: {x, y, z}",
    "pagination": {
      "nextSelector":  "li.next > a",
      "maxPages":      3
    }
  }]
}

Field	Type	Required	Description
`observe`	string	Yes	Plain English description of the elements to find and process
`mode`	string	No	How to interact with each element. One of `inline`, `navigate`, or `click`. Defaults to `click`.
`captureSelector`	string	No	CSS selector or plain English description of which part of the page to capture per item
`maxItems`	number	No	Maximum number of elements to process. Default is 50, maximum is 50.
`waitAfterClick`	number	No	Milliseconds to wait after clicking or navigating before capturing. Default is 2500.
`actions`	array	No	Browser actions to run on each item after clicking or navigating, before capturing
`itemPrompt`	string	No	AI extraction prompt to run on each item individually
`pagination.nextSelector`	string	Yes if using pagination	CSS selector or plain English description of the next page button
`pagination.maxPages`	number	No	How many additional pages to process beyond the first. Default is 5, maximum is 10.

Get Started

Features

Dashboard

Billing & Usage

Advanced

Integrations

Browser Actions

What Are Browser Actions?

How to Target Elements

CSS or XPath selectors (the `selector` field)

Plain English descriptions (the `value` field)

Actions Reference

click

type

check

uncheck

wait

scroll

forEach: Process Every Element on a Page

Adding forEach

Writing a good observe instruction

Do you actually need forEach?

The three forEach modes

inline mode: Read the element directly

navigate mode: Follow each link to its destination page

click mode: Click to expand, capture, then move on

captureSelector

itemPrompt: Extract specific fields from each item

Per-element actions: Do something on each item before capturing

maxItems and waitAfterClick

Advanced Patterns

Pattern 1: Click to a category first, then forEach over its items

Pattern 3: Scrape multiple categories at the same time

Pattern 4: Click to category, navigate each book, scroll to load full description

Response Format

itemPrompt vs Top-level prompt

Limits

Full forEach Field Reference

Get Started

Features

Dashboard

Billing & Usage

Advanced

Integrations

​What Are Browser Actions?

​How to Target Elements

​CSS or XPath selectors (the selector field)

​Plain English descriptions (the value field)

​Actions Reference

​click

​type

​check

​uncheck

​wait

​scroll

​forEach: Process Every Element on a Page

​Adding forEach

​Writing a good observe instruction

​Do you actually need forEach?

​The three forEach modes

​inline mode: Read the element directly

​navigate mode: Follow each link to its destination page

​click mode: Click to expand, capture, then move on

​captureSelector

​itemPrompt: Extract specific fields from each item

​pagination: Keep going to the next page

​Per-element actions: Do something on each item before capturing

​maxItems and waitAfterClick

​Advanced Patterns

​Pattern 1: Click to a category first, then forEach over its items

​Pattern 2: Click to a category using plain English, then forEach with pagination

​Pattern 3: Scrape multiple categories at the same time

​Pattern 4: Click to category, navigate each book, scroll to load full description

​Response Format

​itemPrompt vs Top-level prompt

​Limits

​Full forEach Field Reference

What Are Browser Actions?

How to Target Elements

CSS or XPath selectors (the `selector` field)

Plain English descriptions (the `value` field)

Actions Reference

click

type

check

uncheck

wait

scroll

forEach: Process Every Element on a Page

Adding forEach

Writing a good observe instruction

Do you actually need forEach?

The three forEach modes

inline mode: Read the element directly

navigate mode: Follow each link to its destination page

click mode: Click to expand, capture, then move on

captureSelector

itemPrompt: Extract specific fields from each item

pagination: Keep going to the next page

Per-element actions: Do something on each item before capturing

maxItems and waitAfterClick

Advanced Patterns

Pattern 1: Click to a category first, then forEach over its items

Pattern 2: Click to a category using plain English, then forEach with pagination

Pattern 3: Scrape multiple categories at the same time

Pattern 4: Click to category, navigate each book, scroll to load full description

Response Format

itemPrompt vs Top-level prompt

Limits

Full forEach Field Reference