Structured output

With only a prompt, the AI decides the shape of the answer: field names, types and nesting can change between pages and runs, and prices come back as text like "£51.77". Add a schema and every result has exactly the fields and types you define. Use one whenever code reads the result. Schemas work on Scrape, Batch scrape, Crawl and Parse.

A first schema

{
  "urls": [{ "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" }],
  "prompt": "Extract the book details",
  "schema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "price": { "type": "number" },
      "currency": { "type": ["string", "null"] },
      "in_stock": { "type": ["boolean", "null"] },
      "rating": { "type": ["string", "null"] }
    }
  }
}

result.content:

{ "title": "A Light in the Attic", "price": 51.77, "currency": "£", "in_stock": true, "rating": null }

price is the number 51.77; the £ moved into currency.
in_stock is a real boolean, not the text “In stock”.
rating is null: the site shows its stars as an image, and Spidra reads the page’s text.

It cost 2 credits: 1 for the page and 1 for the AI.

Every field comes back

Every field in your schema is in the result. When the page doesn’t have a value, the field is null; it’s never left out. On a book page with no ISBN:

{
  "type": "object",
  "properties": {
    "title":  { "type": "string" },
    "price":  { "type": "number" },
    "isbn":   { "type": ["string", "null"] },
    "author": { "type": ["string", "null"] }
  }
}

{ "title": "A Light in the Attic", "price": 51.77, "isbn": null, "author": "Shel Silverstein" }

The answer was the same whether required listed the fields or not, so code can always read every key. Give fields that may be missing a nullable type, as below, so null is a valid value.

Nullable fields

Write the type as a list that includes "null":

{ "type": ["string", "null"] }

This works for every type: ["number", "null"], ["boolean", "null"], ["array", "null"].

Lists of items

The root of a schema is always an object. To get a list, put an array inside it:

{
  "urls": [{ "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" }],
  "prompt": "Every book on the page",
  "schema": {
    "type": "object",
    "properties": {
      "books": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "title": { "type": "string" },
            "price": { "type": "number" },
            "in_stock": { "type": ["boolean", "null"] }
          }
        }
      }
    }
  }
}

That returned all 20 books on the page. The first three:

{
  "books": [
    { "title": "Sharp Objects", "price": 47.82, "in_stock": true },
    { "title": "In a Dark, Dark Wood", "price": 19.63, "in_stock": true },
    { "title": "The Past Never Ends", "price": 56.5, "in_stock": true }
  ]
}

A flat schema (no array) on a list page returns one item. In a crawl, the schema runs on every returned page, so write it for one page of the kind you asked for.

Fixed choices with `enum`

enum limits a field to the values you list. Include null for “none of these”:

{
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "stars": { "type": ["string", "null"], "enum": ["one", "two", "three", "four", "five", null] },
    "availability": { "type": ["string", "null"], "enum": ["in_stock", "out_of_stock", null] }
  }
}

{ "title": "A Light in the Attic", "stars": null, "availability": "in_stock" }

stars is null because the rating isn’t in the page’s text.

Nested objects

Define properties inside a property:

{
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "price": {
      "type": "object",
      "properties": {
        "amount":   { "type": ["number", "null"] },
        "currency": { "type": ["string", "null"] }
      }
    }
  }
}

{ "title": "A Light in the Attic", "price": { "amount": 51.77, "currency": "£" } }

Prompt and schema together

The schema sets the shape; the prompt tells the AI what to pick and how to write it. Here the prompt filters the quotes and lowercases the tags:

{
  "urls": [{ "url": "https://quotes.toscrape.com/" }],
  "prompt": "Only the quotes by Albert Einstein. Give the tags in lowercase.",
  "schema": {
    "type": "object",
    "properties": {
      "quotes": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": { "text": { "type": "string" }, "tags": { "type": "array", "items": { "type": "string" } } }
        }
      }
    }
  }
}

{
  "quotes": [
    { "text": "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.", "tags": ["change", "deep-thoughts", "thinking", "world"] },
    { "text": "There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.", "tags": ["inspirational", "life", "live", "miracle", "miracles"] }
  ]
}

(Shortened to two of the Einstein quotes on the page.)

Generate a schema from Zod or Pydantic

Spidra takes standard JSON Schema, so you can generate it from the models you already have.

import { z } from "zod";

const Book = z.object({
  title: z.string(),
  price: z.number(),
  in_stock: z.boolean().nullable(),
});

const result = await spidra.scrape({
  urls: [{ url: "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" }],
  prompt: "Extract the book details",
  schema: z.toJSONSchema(Book),
});

const book = Book.parse(result.content);

from typing import Optional
from pydantic import BaseModel
from spidra import ScrapeParams, ScrapeUrl

class Book(BaseModel):
    title: str
    price: float
    in_stock: Optional[bool] = None

result = client.scrape(ScrapeParams(
    urls=[ScrapeUrl(url="https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html")],
    prompt="Extract the book details",
    schema=Book.model_json_schema(),
))

book = Book.model_validate(result.content)

Validating the result with the same model gives you typed data end to end. There’s also a visual JSON Schema generator.

What a schema can contain

Supported keywords: type (including lists with "null"), properties, required, items, enum, description and nullable. A description on a field is a good place for hints (“price without the currency symbol”). Keywords Spidra can’t use, such as $ref, oneOf, allOf, not and if, are ignored, and the submit response lists them in schema_warnings:

{
  "status": "queued",
  "jobId": "536b5dd6-84e7-4287-b514-5c7250750695",
  "schema_warnings": ["\"anyOf\" at schema.properties.isbn is not supported and will be ignored"]
}

The job still runs. Pydantic writes Optional fields as anyOf with null, which shows this warning; those fields still come back, as null when missing.

Schema errors

A schema with a structural problem is refused with 422 before anything runs, and nothing is charged:

{
  "status": "error",
  "message": "Invalid schema. Fix the errors below and try again.",
  "errors": ["schema is too deeply nested (depth: 6). Maximum nesting depth is 5. Flatten your schema."]
}

The root type must be "object". Wrap lists in an object property, as above.
Nesting goes 5 levels deep at most.
The schema can be up to 10 KB.

Sites that come back structured already

Links from supported sites such as Amazon, Google Maps and LinkedIn have the site’s own fields in structuredData, and with "output": "json" and no prompt or schema those fields are the content, with no AI. Add a schema only if you want to reshape them.

Get started

Tools

Ready-made data

Getting the data you want

Supported sites

Guides

Dashboard

Account and billing

Structured output

A first schema

Every field comes back

Nullable fields

Lists of items

Fixed choices with `enum`

Nested objects

Prompt and schema together

Generate a schema from Zod or Pydantic

What a schema can contain

Schema errors

Sites that come back structured already

​A first schema

​Every field comes back

​Nullable fields

​Lists of items

​Fixed choices with enum

​Nested objects

​Prompt and schema together

​Generate a schema from Zod or Pydantic

​What a schema can contain

​Schema errors

​Sites that come back structured already

A first schema

Every field comes back

Nullable fields

Lists of items

Fixed choices with `enum`

Nested objects

Prompt and schema together

Generate a schema from Zod or Pydantic

What a schema can contain

Schema errors

Sites that come back structured already