# AI Data Extraction: Any Web Page to JSON Schema (`fernandosord/ai-structured-extractor`) Actor

Extract structured data from a list of URLs with an LLM. Give a JSON Schema or a list of fields and get one JSON record per page, validated against the schema and retried once on failure. Uses your Anthropic, OpenAI or OpenRouter key, with a key-free demo. About $0.01 per page plus your LLM cost.

- **URL**: https://apify.com/fernandosord/ai-structured-extractor.md
- **Developed by:** [Plainsource](https://apify.com/fernandosord) (community)
- **Categories:** AI, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Structured Data Extractor

Give this Actor a list of public web pages and describe the data you want, either as plain field names or as a JSON Schema. It reads each page, asks a language model to fill in your fields, checks the answer against the schema, and saves one result per page. You use your own Anthropic, OpenAI or OpenRouter API key, so the LLM cost goes to your provider account at their rates.

Without a key it runs a demo that needs no LLM: it returns each page's title, meta description, headings and links. That is what the default input does.

### What you get for each page

- `data`: an object that matches your schema, or an array of objects for list pages such as category or search pages.
- `status`: `success` only when `data` passed schema validation. Anything else comes with a reason in `error` or `validationErrors`.
- The URL after redirects, whether a browser was needed, how many LLM calls were made, and how many tokens your provider billed.

### How a page is processed

- The page is fetched with a plain HTTP request.
- If the response is an empty JavaScript shell (little visible text plus a script-driven root element), it is loaded once more in headless Chromium. You can turn this off with **Render JavaScript pages in a browser when needed**.
- Scripts, styles, hidden elements and anything matching **Remove CSS selector** are dropped. The rest is converted to markdown with absolute links and image URLs. **Content CSS selector** can narrow it to one part of the page.
- The markdown and your schema go to the model. The page text is marked as untrusted data in the prompt, so instructions hidden in a page are not followed.
- The reply is parsed and validated with [Ajv](https://ajv.js.org/). If it is not valid JSON or fails the schema, the model gets the exact validation errors and one more try. If the second answer still fails, the row is saved with `status: "validation_failed"` and is not charged.

### Quick start

1. Run the Actor with the default input and no key. You get three demo rows in a few seconds.
2. Paste your API key into **LLM API key** and choose the matching **LLM provider**.
3. List your fields, one per line, for example:

```text
title
price: price as shown on the page, including the currency symbol
availability: stock status text
```

The part before the colon becomes the JSON key and the rest tells the model what you mean. With plain fields every key is required, extra keys are rejected, and a value the page does not show comes back as `null`.

For a page with many items, set **Page type** to `list` and each row's `data` becomes an array with one object per item.

### Using a JSON Schema

For typed values, enums or nested objects, put a JSON Schema into **JSON Schema**. It replaces the plain fields. This one extracts every quote on a listing page:

```json
{
  "type": "array",
  "items": {
    "type": "object",
    "required": ["text", "author", "tags"],
    "additionalProperties": false,
    "properties": {
      "text": { "type": "string" },
      "author": { "type": "string" },
      "tags": { "type": "array", "items": { "type": "string" } }
    }
  }
}
```

Validation follows JSON Schema draft-07, including `format` values such as `uri`, `email` and `date`. A `$schema` key, if present, is ignored.

### Input

| Field | Input key | Default | Notes |
|---|---|---|---|
| URLs | `startUrls` | three sample pages | Required. Duplicates are removed. You can also upload a text file of URLs. |
| Fields (plain language) | `fields` | `title`, `price`, `availability` | Ignored when a JSON Schema is set. |
| Page type | `pageType` | `single` | `list` returns an array per page. Applies to plain fields only. |
| JSON Schema | `jsonSchema` | none | Overrides `fields`. |
| Extra instructions | `instructions` | none | Free text for the model, e.g. units or date format. |
| LLM provider | `llmProvider` | `anthropic` | `anthropic`, `openai` or `openrouter`. |
| LLM API key | `llmApiKey` | none | Secret field. Empty means demo mode. |
| Model | `llmModel` | provider default | `claude-haiku-5-5`, `gpt-5.4-mini` or `anthropic/claude-haiku-5.5`. Any model ID your provider accepts works. |
| Max page characters sent to the LLM | `maxContentChars` | 60000 | Longer pages are cut and marked `truncated: true`. |
| Content CSS selector | `contentCssSelector` | none | Send only this part of the page. |
| Remove CSS selector | `removeCssSelector` | none | Drop these elements first. |
| Render JavaScript pages in a browser when needed | `useBrowserFallback` | `true` | |
| Max pages | `maxPages` | 100 | |
| Max concurrency | `maxConcurrency` | 5 | Lower it if your LLM account hits rate limits. |
| Extra domains to skip | `extraBlockedDomains` | none | Added to the built-in list below. |

### Output

One dataset row per input URL, including the ones that failed or were skipped, so you can see what happened to every URL. The **Overview** view shows status, data and errors; **Failed or skipped pages** shows only the diagnostic columns.

This row is from a demo-mode run of the default input on 2026-10-21 (link list shortened to three entries):

```json
{
  "url": "https://quotes.toscrape.com/",
  "loadedUrl": "https://quotes.toscrape.com/",
  "status": "success",
  "mode": "demo",
  "fetchedWith": "http",
  "httpStatus": 200,
  "data": {
    "title": "Quotes to Scrape",
    "metaDescription": null,
    "canonicalUrl": null,
    "language": "en",
    "h1": ["Quotes to Scrape"],
    "h2": ["Top Ten tags"],
    "wordCount": 278,
    "linkCount": 49,
    "links": [
      { "text": "Quotes to Scrape", "url": "https://quotes.toscrape.com/" },
      { "text": "Login", "url": "https://quotes.toscrape.com/login" },
      { "text": "(about)", "url": "https://quotes.toscrape.com/author/Albert-Einstein" }
    ]
  },
  "validationErrors": [],
  "error": null,
  "attempts": 0,
  "contentChars": null,
  "truncated": false,
  "provider": null,
  "model": null,
  "usage": null,
  "note": "Demo mode: no LLM API key was given, so this row holds page facts read directly from the HTML (title, meta, headings, links). Add an API key to extract your own fields.",
  "extractedAt": "2026-10-21T05:09:22.918Z"
}
```

In LLM mode the row has the same keys, without `note`. `data` holds your fields, and `provider`, `model`, `attempts`, `contentChars` and `usage` are filled in.

| Key | Meaning |
|---|---|
| `url` | URL as given in the input |
| `loadedUrl` | Final URL after redirects |
| `status` | See the table below |
| `mode` | `llm` or `demo` |
| `fetchedWith` | `http` or `browser` |
| `httpStatus` | Status code of the page response |
| `data` | The extracted object or array, or the demo shape |
| `validationErrors` | Schema errors left after the retry, e.g. `/price must be number` |
| `error` | Reason for a skip or failure |
| `attempts` | LLM calls made for the page: 1, or 2 after a retry |
| `contentChars` | Length of the markdown sent to the model |
| `truncated` | `true` if the page was cut at `maxContentChars` |
| `provider`, `model` | Which LLM produced the row |
| `usage` | `inputTokens` and `outputTokens` billed by your provider for this page, retries included |
| `note` | Demo mode only |
| `extractedAt` | ISO timestamp |

| `status` | Charged | What happened |
|---|---|---|
| `success` | yes, in LLM mode | `data` matches the schema |
| `validation_failed` | no | Two answers in a row failed the schema; the last one is in `data` |
| `llm_error` | no | Provider error, refusal, or output cut off. A rejected key or unknown model stops the run. |
| `fetch_failed` | no | HTTP error, timeout, non-HTML content, or page over 8 MB |
| `no_content` | no | Nothing readable left after cleanup |
| `blocked_domain` | no | On the blocked list, or redirected there |
| `not_allowed` | no | The URL, or a redirect, points to a private or local address (localhost, internal networks, cloud metadata) |
| `robots_disallowed` | no | The site's robots.txt disallows it |
| `invalid_url` | no | Not an http(s) URL |
| `skipped` | no | The run stopped first (your cost limit was reached, or the key was rejected) |

A `SUMMARY` record in the run's key-value store has the count of each status.

### How much it costs

There are two separate costs.

- **This Actor** uses pay-per-event pricing: about $0.01 per `page-extracted` event plus Apify's small per-run start fee. The Pricing tab shows the current figures. A page is charged only when its result passed schema validation. Demo runs, failures and skipped pages are never charged per page.
- **Your LLM provider** bills your own account for tokens. The `usage` field shows the exact numbers per page. As a rough guide, at about 4 characters per token, a 20,000-character page is about 5,000 input tokens. With `claude-haiku-5-5` at $0.10 per million input tokens and $0.50 per million output tokens (Anthropic's list price on 2026-10-21), that is about $0.0005 of input plus the reply. Larger models cost more.

If you set a maximum cost for the run, the Actor checks the remaining budget before each LLM call. Once the budget is used up, the remaining pages are saved as `skipped` without being sent to your provider.

### Blocked sites and robots.txt

These are always skipped because their terms forbid automated collection and their owners enforce that, in some cases in court: linkedin.com, facebook.com, instagram.com, threads.net, amazon (all country domains), x.com, twitter.com, youtube.com, tiktok.com, and Google search result pages (`google.*/search`). Other Google pages, such as developer docs, are not blocked. You can add more hosts with **Extra domains to skip**, but you cannot remove these.

Only public addresses are fetched. URLs that point to localhost, private networks, link-local or cloud metadata addresses, or hostnames that resolve to them, are saved with status `not_allowed` and never requested. Every redirect, and every request the headless browser makes, is checked the same way.

Every URL is checked against its site's robots.txt for the user agent `AIStructuredExtractor`. Disallowed URLs are saved with status `robots_disallowed` and not fetched. If a site's robots.txt returns a server error or times out, the site is treated as disallowed.

The Actor does not log in, solve captchas, rotate proxies or change its user agent to get past a block. Requests use one identifiable user agent: `Mozilla/5.0 (compatible; AIStructuredExtractor/0.1; +https://apify.com/store)`.

### Your API key

The key field is an Apify secret input, so Apify stores it encrypted along with the rest of the run input. The Actor sends it only to the provider you chose. It never logs the key or writes it to the dataset or to any record of its own, and it strips the key from provider error messages before they are logged.

### Limits

- Pages behind a login, a paywall or bot protection fail with `fetch_failed`. There is no workaround on purpose.
- JavaScript detection is a heuristic. A page that ships some server-rendered text but loads the data you want later by script may not trigger the browser. Use a narrower **Content CSS selector** and check `contentChars` if results look empty.
- The browser fallback waits for the network to go quiet for up to 15 seconds. Infinite scroll, "load more" buttons and content behind clicks are not handled.
- Only HTML and plain-text pages are read. PDFs, images and other files fail with `fetch_failed`.
- Long pages are cut at `maxContentChars`. Items past the cut are missing from list pages, and the row says `truncated: true`.
- The model can still misread a page. Schema validation catches wrong types, missing keys and extra keys, but it cannot tell whether a correctly typed value is the right one. Spot-check results before relying on them.
- Accuracy and cost depend on the model you choose. Small, cheap models are more likely to skip items on long lists, mix up rows in dense tables, or fill a field from the wrong part of the page. If that happens, try a larger model through **Model**.
- One URL gives one row. The Actor does not follow links or crawl a site.

### Running it locally

```bash
npm install
npx playwright install chromium
npx apify-cli run
```

With no key it runs in demo mode. To test LLM mode locally without writing the key into `storage/key_value_stores/default/INPUT.json`, export it as `LLM_API_KEY` in your shell before `npx apify-cli run`. This environment fallback is ignored on the Apify platform.

`npm test` runs the unit tests. `test/mock-llm.mjs` is a local stand-in for the Anthropic and OpenAI APIs, used to test retries, charging and error handling without spending tokens.

### Feedback

Open an issue on the Issues tab with the URL, your schema and the run ID. Please leave your API key out.

# Changelog

This Actor's version history is a separate document: https://apify.com/fernandosord/ai-structured-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Public web pages to extract from. Each URL becomes one row in the dataset. Duplicates are removed.

## `fields` (type: `array`):

One field per line, optionally followed by a colon and a description, e.g. `price: listed price including currency`. Ignored when a JSON Schema is given.

## `pageType` (type: `string`):

`single`: one object per page, e.g. a product page. `list`: an array with one object per item, e.g. a search result or category page. With a JSON Schema, use a root of type `array` instead.

## `jsonSchema` (type: `object`):

A JSON Schema for the result of each page. When set, it is used instead of the plain-language fields. Results that do not validate are retried once and then saved with status `validation_failed`.

## `instructions` (type: `string`):

Optional notes for the model, e.g. `Prices are in EUR; return numbers without the currency sign.`

## `llmProvider` (type: `string`):

Which API your key belongs to.

## `llmApiKey` (type: `string`):

Your API key for the chosen provider. Stored encrypted by Apify and never logged or written to the dataset. Leave empty to run the key-free demo.

## `llmModel` (type: `string`):

Model ID. Leave empty for the default: `claude-haiku-5-5` (Anthropic), `gpt-5.4-mini` (OpenAI), `anthropic/claude-haiku-5.5` (OpenRouter).

## `maxContentChars` (type: `integer`):

The page is converted to markdown first. Longer pages are cut at this length and the row is marked `truncated: true`. Lower it to cut LLM cost.

## `contentCssSelector` (type: `string`):

Optional. Only send the matching part of the page to the LLM, e.g. `main` or `#product`. Falls back to the whole body if nothing matches.

## `removeCssSelector` (type: `string`):

Optional. Elements to drop before extraction, e.g. `nav, footer, .cookie-banner`.

## `useBrowserFallback` (type: `boolean`):

Pages are fetched with plain HTTP first. If the response is an empty JavaScript shell, the page is loaded once more in headless Chromium.

## `maxPages` (type: `integer`):

Stop after this many URLs.

## `maxConcurrency` (type: `integer`):

How many pages are processed at the same time. Lower it if your LLM account hits rate limits.

## `extraBlockedDomains` (type: `array`):

Hostnames to skip in addition to the built-in list (subdomains included), e.g. `example.org`.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
    },
    {
      "url": "https://quotes.toscrape.com/"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping"
    }
  ],
  "fields": [
    "title",
    "price: price as shown on the page, including the currency symbol",
    "availability: stock status text"
  ],
  "pageType": "single",
  "llmProvider": "anthropic",
  "maxContentChars": 60000,
  "useBrowserFallback": true,
  "maxPages": 100,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One JSON record per page, with validation status.

## `summary` (type: `string`):

Counts of successful, failed and skipped pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
        },
        {
            "url": "https://quotes.toscrape.com/"
        },
        {
            "url": "https://en.wikipedia.org/wiki/Web_scraping"
        }
    ],
    "fields": [
        "title",
        "price: price as shown on the page, including the currency symbol",
        "availability: stock status text"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fernandosord/ai-structured-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" },
        { "url": "https://quotes.toscrape.com/" },
        { "url": "https://en.wikipedia.org/wiki/Web_scraping" },
    ],
    "fields": [
        "title",
        "price: price as shown on the page, including the currency symbol",
        "availability: stock status text",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("fernandosord/ai-structured-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
    },
    {
      "url": "https://quotes.toscrape.com/"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping"
    }
  ],
  "fields": [
    "title",
    "price: price as shown on the page, including the currency symbol",
    "availability: stock status text"
  ]
}' |
apify call fernandosord/ai-structured-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fernandosord/ai-structured-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vCvOj8yTGPMaccIm6/builds/DugyUjLT0qYUsgCqE/openapi.json