> ## Documentation Index
> Fetch the complete documentation index at: https://docs.spidra.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Go

> Official Go SDK for the Spidra web scraping API. Fully typed structs, zero external dependencies. Scrape, batch, crawl, and run browser actions from Go.

The Go SDK wraps the Spidra API with fully typed request and response structs. No `map[string]any` to wrestle with — you get concrete types for everything, errors as values, and zero external dependencies. Just the standard library.

## Installation

```bash theme={null}
go get github.com/spidra-io/spidra-go
```

<Note>
  Get your API key from [app.spidra.io](https://app.spidra.io) under **Settings → API Keys**.
  Store it as an environment variable — never hardcode it in source.
</Note>

## Getting started

```go theme={null}
import spidra "github.com/spidra-io/spidra-go"

client := spidra.New(os.Getenv("SPIDRA_API_KEY"))
```

If you need to override the base URL (for local dev or self-hosted) or set a custom HTTP timeout:

```go theme={null}
client := spidra.New(
    os.Getenv("SPIDRA_API_KEY"),
    spidra.WithBaseURL("http://localhost:4321/api"),
    spidra.WithTimeout(30 * time.Second),
)
```

Everything lives under `client.Scrape`, `client.Batch`, `client.Crawl`, `client.Logs`, and `client.Usage`.

## Scraping

Each scrape request can include up to three URLs and runs them in parallel. You tell the AI what to extract through a `Prompt`, and optionally lock the output shape with a `Schema`.

The quickest path is `Run()` — it submits the job, polls until it completes, and returns the full result:

```go theme={null}
job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{
    URLs: []spidra.ScrapeURL{
        {URL: "https://example.com/pricing"},
    },
    Prompt: "Extract all pricing plans with name, price, and included features",
    Output: "json",
})
if err != nil {
    return err
}

fmt.Println(job.Result.Content)
```

When you need more control — say you're building a queue and want to poll on your own schedule — use `Submit()` and `Get()` separately:

```go theme={null}
job, err := client.Scrape.Submit(ctx, spidra.ScrapeParams{
    URLs:   []spidra.ScrapeURL{{URL: "https://example.com"}},
    Prompt: "Extract the main headline",
})
if err != nil {
    return err
}

// Come back later and check
status, err := client.Scrape.Get(ctx, job.JobID)
if err != nil {
    return err
}

if status.Status == "completed" {
    fmt.Println(status.Result.Content)
}
```

Jobs move through `waiting` → `active` → `completed` (or `failed`).

### ScrapeParams fields

| Field                | Type          | Description                                                     |
| -------------------- | ------------- | --------------------------------------------------------------- |
| `URLs`               | `[]ScrapeURL` | Up to 3 URLs. Each can carry its own `Actions` slice            |
| `Prompt`             | `string`      | What to extract, in plain English                               |
| `Output`             | `string`      | `"markdown"` (default) or `"json"`                              |
| `Schema`             | `any`         | JSON Schema object — locks the output shape when using `"json"` |
| `UseProxy`           | `bool`        | Route through a residential proxy                               |
| `ProxyCountry`       | `string`      | Two-letter country code: `"us"`, `"de"`, `"jp"`, etc.           |
| `ExtractContentOnly` | `bool`        | Strip nav, ads, and boilerplate before the AI sees the page     |
| `Screenshot`         | `bool`        | Capture a viewport screenshot                                   |
| `FullPageScreenshot` | `bool`        | Capture a full-page (scrolled) screenshot                       |
| `Cookies`            | `string`      | Raw `Cookie` header string for pages behind a login             |

### Enforcing an exact output shape

Pass a `Schema` when you need the output to match a specific structure. Fields the AI can't find come back as `null` instead of guessed values:

```go theme={null}
job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{
    URLs:   []spidra.ScrapeURL{{URL: "https://jobs.example.com/senior-engineer"}},
    Prompt: "Extract the job listing details",
    Output: "json",
    Schema: map[string]any{
        "type":     "object",
        "required": []string{"title", "company", "remote"},
        "properties": map[string]any{
            "title":      map[string]any{"type": "string"},
            "company":    map[string]any{"type": "string"},
            "remote":     map[string]any{"type": []string{"boolean", "null"}},
            "salary_min": map[string]any{"type": []string{"number", "null"}},
            "skills":     map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
        },
    },
})
```

### Geo-targeted scraping

Some sites return different prices or content based on the visitor's location. Set `UseProxy: true` and pick a `ProxyCountry` to route through a residential IP in that region:

```go theme={null}
job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{
    URLs:         []spidra.ScrapeURL{{URL: "https://www.amazon.de/gp/bestsellers"}},
    Prompt:       "List the top 10 products with name and price",
    UseProxy:     true,
    ProxyCountry: "de",
})
```

Supported codes include `us`, `gb`, `de`, `fr`, `jp`, `au`, `ca`, `br`, `in`, `nl`, `sg`, and [40+ more](/features/stealth-mode#country-targeting). Use `"global"` or `"eu"` for regional routing without pinning to a specific country.

### Scraping pages behind a login

Pass your session cookies as a raw header string. The easiest way to grab this is to open browser devtools, log in, and copy the `Cookie` header from any authenticated request:

```go theme={null}
job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{
    URLs:    []spidra.ScrapeURL{{URL: "https://app.example.com/dashboard"}},
    Prompt:  "Extract the monthly revenue and active user count",
    Cookies: "session=abc123; auth_token=xyz789",
})
```

### Browser actions

For pages that need interaction before you can extract anything — accepting cookies, typing into a search input, scrolling to trigger lazy loading — include an `Actions` slice on the URL. They run in order before the AI sees the page:

```go theme={null}
job, err := client.Scrape.Run(ctx, spidra.ScrapeParams{
    URLs: []spidra.ScrapeURL{
        {
            URL: "https://example.com/products",
            Actions: []map[string]any{
                {"type": "click", "selector": "#accept-cookies"},
                {"type": "wait", "duration": 1000},
                {"type": "scroll", "to": "80%"},
            },
        },
    },
    Prompt: "Extract all product names and prices visible on the page",
})
```

For `selector` pass a CSS selector or XPath. If you'd rather describe the element in words, use `value` — Spidra will locate it using AI.

| Action    | What it does                                                            |
| --------- | ----------------------------------------------------------------------- |
| `click`   | Click any element — CSS selector via `selector`, plain text via `value` |
| `type`    | Type into an input or textarea                                          |
| `check`   | Check a checkbox                                                        |
| `uncheck` | Uncheck a checkbox                                                      |
| `wait`    | Pause for `duration` milliseconds                                       |
| `scroll`  | Scroll to a percentage of page height (e.g. `"80%"`)                    |
| `forEach` | Loop over every matched element and process each one                    |

### Controlling how long Run() waits

By default `Run()` polls every 3 seconds and times out after 120 seconds. Pass `PollOptions` to override:

```go theme={null}
job, err := client.Scrape.Run(ctx, params, spidra.PollOptions{
    PollInterval: 5 * time.Second,
    Timeout:      60 * time.Second,
})
```

On timeout, `Run()` returns the result with `Status: "timeout"` rather than an error — the `JobID` is still there so you can keep polling with `Get()` if you need to wait longer. The same options work on `Batch.Run()` and `Crawl.Run()`.

## Batch scraping

When you have a list of URLs to process, batch is the right tool. Submit up to 50 URLs in a single request and they run in parallel. Unlike the scraper, each URL here is a plain string — no per-URL actions.

```go theme={null}
batch, err := client.Batch.Run(ctx, spidra.BatchParams{
    URLs: []string{
        "https://shop.example.com/product/1",
        "https://shop.example.com/product/2",
        "https://shop.example.com/product/3",
    },
    Prompt:   "Extract product name, price, and whether it is in stock",
    Output:   "json",
    UseProxy: true,
})
if err != nil {
    return err
}

fmt.Printf("%d/%d completed\n", batch.CompletedCount, batch.TotalURLs)

for _, item := range batch.Items {
    if item.Status == "completed" {
        fmt.Println(item.URL, item.Result)
    } else if item.Status == "failed" {
        fmt.Println("failed:", item.URL, item.Error)
    }
}
```

Each item moves through `pending` → `running` → `completed` (or `failed`). The batch as a whole follows the same lifecycle plus a `cancelled` state.

For fire-and-forget, use `Submit()` and come back with `Get()`:

```go theme={null}
batch, err := client.Batch.Submit(ctx, spidra.BatchParams{
    URLs:   []string{"https://example.com/1", "https://example.com/2"},
    Prompt: "Extract the page title and meta description",
})
if err != nil {
    return err
}

// Later...
result, err := client.Batch.Get(ctx, batch.BatchID)
fmt.Printf("%s: %d/%d done\n", result.Status, result.CompletedCount, result.TotalURLs)
```

### Retrying failures

If some items fail, you can re-queue just those without touching the ones that already succeeded:

```go theme={null}
result, _ := client.Batch.Get(ctx, batchID)

if result.FailedCount > 0 {
    if err := client.Batch.Retry(ctx, batchID); err != nil {
        return err
    }
}
```

To stop a running batch and get credits back for anything that hasn't started:

```go theme={null}
if err := client.Batch.Cancel(ctx, batchID); err != nil {
    return err
}
```

To browse past batches:

```go theme={null}
page, err := client.Batch.List(ctx, 1, 20) // page, limit

for _, job := range page.Jobs {
    fmt.Printf("%s %s — %d/%d\n", job.UUID, job.Status, job.CompletedCount, job.TotalURLs)
}
```

***

## Crawling

Crawling is for when you need to cover a whole site or section, not just a handful of URLs. You give it a starting URL and instructions for which links to follow; it discovers pages on its own, extracts data from each one, and hands everything back when it's done.

```go theme={null}
job, err := client.Crawl.Run(ctx, spidra.CrawlParams{
    BaseURL:              "https://competitor.com/blog",
    CrawlInstruction:     "Follow links to blog posts only — skip tag pages, category pages, and the homepage",
    TransformInstruction: "Extract the post title, author name, publish date, and a one-sentence summary",
    MaxPages:             30,
    UseProxy:             true,
})
if err != nil {
    return err
}

for _, page := range job.Result {
    fmt.Println(page.URL, page.Data)
}
```

`CrawlInstruction` controls navigation — which links to follow, which to skip. `TransformInstruction` controls extraction — what to pull from each page. `MaxPages` is a cap so a crawl doesn't run forever.

The same `UseProxy`, `ProxyCountry`, and `Cookies` options from the scraper all work here.

For fire-and-forget:

```go theme={null}
job, err := client.Crawl.Submit(ctx, spidra.CrawlParams{
    BaseURL:              "https://example.com/docs",
    CrawlInstruction:     "Follow all documentation pages",
    TransformInstruction: "Extract the page title and a short summary",
    MaxPages:             50,
})

// Poll later
status, err := client.Crawl.Get(ctx, job.JobID)
// status.Status: "waiting" | "active" | "running" | "completed" | "failed"
```

### Downloading the raw content

Once a crawl is complete, you can get signed URLs to download the raw HTML and Markdown for every page that was visited. Links expire after an hour:

```go theme={null}
result, err := client.Crawl.Pages(ctx, jobID)

for _, page := range result.Pages {
    fmt.Println(page.URL, page.Status)
    // page.HTMLURL     — download the raw HTML
    // page.MarkdownURL — download the cleaned Markdown
}
```

### Re-extracting with a different prompt

Crawling is the expensive part. If you've already crawled a site and just want to pull out different information, you don't have to crawl again — `Extract()` runs a new AI pass over the already-stored content and only charges transformation credits:

```go theme={null}
result, err := client.Crawl.Extract(
    ctx,
    completedJobID,
    "Extract product SKUs and prices as structured JSON",
)
if err != nil {
    return err
}

// This creates a new job — poll it like any other
extracted, err := client.Crawl.Get(ctx, result.JobID)
```

### History and stats

```go theme={null}
history, err := client.Crawl.History(ctx, 1, 10) // page, limit
fmt.Printf("%d total crawl jobs\n", history.Total)

stats, err := client.Crawl.Stats(ctx)
fmt.Printf("%d all-time crawls\n", stats.Total)
```

## Logs

Every scrape request your API key makes is logged automatically. You can query the full history and filter by status, URL, date range, or channel:

```go theme={null}
result, err := client.Logs.List(ctx, map[string]string{
    "status":     "failed",
    "searchTerm": "amazon.com",
    "dateStart":  "2024-01-01",
    "dateEnd":    "2024-12-31",
    "page":       "1",
    "limit":      "20",
})
if err != nil {
    return err
}

for _, entry := range result.Logs {
    fmt.Println(entry.URLs[0].URL, entry.Status, entry.CreditsUsed)
}
```

To get the full details of a single log — including the AI output from that job:

```go theme={null}
entry, err := client.Logs.Get(ctx, logUUID)
fmt.Println(entry.ResultData)
```

## Usage statistics

Check how many requests and credits your account has consumed over a period:

```go theme={null}
result, err := client.Usage.Get(ctx, "30d") // "7d" | "30d" | "weekly"
if err != nil {
    return err
}

for _, row := range result.Data {
    fmt.Printf("%s: %d requests, %d credits\n", row.Date, row.Requests, row.Credits)
}
```

`"7d"` gives one row per day for the past week. `"30d"` gives the last 30 days. `"weekly"` gives one row per week for the past seven weeks.

## Error handling

All API errors are returned as typed error values. Use `errors.As()` to match the specific type you care about:

```go theme={null}
import "errors"

_, err := client.Scrape.Run(ctx, spidra.ScrapeParams{
    URLs:   []spidra.ScrapeURL{{URL: "https://example.com"}},
    Prompt: "Extract the main headline",
})
if err != nil {
    var authErr *spidra.AuthenticationError
    var credErr *spidra.InsufficientCreditsError
    var rateErr *spidra.RateLimitError
    var srvErr  *spidra.ServerError
    var apiErr  *spidra.SpidraError

    switch {
    case errors.As(err, &authErr):
        log.Fatal("invalid or missing API key")
    case errors.As(err, &credErr):
        log.Fatal("account is out of credits")
    case errors.As(err, &rateErr):
        log.Fatal("rate limited — slow down")
    case errors.As(err, &srvErr):
        log.Printf("server error — safe to retry: %s", srvErr.Message)
    case errors.As(err, &apiErr):
        log.Printf("api error %d: %s", apiErr.StatusCode, apiErr.Message)
    default:
        log.Fatal(err)
    }
}
```

| Type                        | Status | When                                |
| --------------------------- | ------ | ----------------------------------- |
| `*AuthenticationError`      | 401    | The API key is missing or invalid   |
| `*InsufficientCreditsError` | 403    | No credits remaining on the account |
| `*RateLimitError`           | 429    | Too many requests — back off        |
| `*ServerError`              | 500    | Unexpected server-side error        |
| `*SpidraError`              | any    | Base type — all others embed this   |

Every error type exposes `StatusCode int` and `Message string`. `*SpidraError` is the base — if you only want one catch-all, match against that.

<CardGroup cols={2}>
  <Card title="PHP" icon="php" href="/sdks/php">
    Official PHP SDK — idiomatic helpers, typed exceptions, and configurable polling.
  </Card>

  <Card title="Ruby" icon="gem" href="/sdks/ruby">
    Official Ruby SDK — pure stdlib, no external dependencies. Works in Rails, Sinatra, and scripts.
  </Card>
</CardGroup>