AI Structured Data Extractor

Give this Actor a list of public web pages and describe the data you want, either as plain field names or as a JSON Schema. It reads each page, asks a language model to fill in your fields, checks the answer against the schema, and saves one result per page. You use your own Anthropic, OpenAI or OpenRouter API key, so the LLM cost goes to your provider account at their rates.

Without a key it runs a demo that needs no LLM: it returns each page's title, meta description, headings and links. That is what the default input does.

What you get for each page

• data: an object that matches your schema, or an array of objects for list pages such as category or search pages.
• status: success only when data passed schema validation. Anything else comes with a reason in error or validationErrors.
• The URL after redirects, whether a browser was needed, how many LLM calls were made, and how many tokens your provider billed.

How a page is processed

• The page is fetched with a plain HTTP request.
• If the response is an empty JavaScript shell (little visible text plus a script-driven root element), it is loaded once more in headless Chromium. You can turn this off with Render JavaScript pages in a browser when needed.
• Scripts, styles, hidden elements and anything matching Remove CSS selector are dropped. The rest is converted to markdown with absolute links and image URLs. Content CSS selector can narrow it to one part of the page.
• The markdown and your schema go to the model. The page text is marked as untrusted data in the prompt, so instructions hidden in a page are not followed.
• The reply is parsed and validated with Ajv. If it is not valid JSON or fails the schema, the model gets the exact validation errors and one more try. If the second answer still fails, the row is saved with status: "validation_failed" and is not charged.

Quick start

1. Run the Actor with the default input and no key. You get three demo rows in a few seconds.
2. Paste your API key into LLM API key and choose the matching LLM provider.
3. List your fields, one per line, for example:

title
price: price as shown on the page, including the currency symbol
availability: stock status text

The part before the colon becomes the JSON key and the rest tells the model what you mean. With plain fields every key is required, extra keys are rejected, and a value the page does not show comes back as null.

For a page with many items, set Page type to list and each row's data becomes an array with one object per item.

Using a JSON Schema

For typed values, enums or nested objects, put a JSON Schema into JSON Schema. It replaces the plain fields. This one extracts every quote on a listing page:

{
  "type": "array",
  "items": {
    "type": "object",
    "required": ["text", "author", "tags"],
    "additionalProperties": false,
    "properties": {
      "text": { "type": "string" },
      "author": { "type": "string" },
      "tags": { "type": "array", "items": { "type": "string" } }
    }
  }
}

Validation follows JSON Schema draft-07, including format values such as uri, email and date. A $schema key, if present, is ignored.

Input

Field	Input key	Default	Notes
URLs	startUrls	three sample pages	Required. Duplicates are removed. You can also upload a text file of URLs.
Fields (plain language)	fields	title, price, availability	Ignored when a JSON Schema is set.
Page type	pageType	single	list returns an array per page. Applies to plain fields only.
JSON Schema	jsonSchema	none	Overrides fields.
Extra instructions	instructions	none	Free text for the model, e.g. units or date format.
LLM provider	llmProvider	anthropic	anthropic, openai or openrouter.
LLM API key	llmApiKey	none	Secret field. Empty means demo mode.
Model	llmModel	provider default	claude-haiku-5-5, gpt-5.4-mini or anthropic/claude-haiku-5.5. Any model ID your provider accepts works.
Max page characters sent to the LLM	maxContentChars	60000	Longer pages are cut and marked truncated: true.
Content CSS selector	contentCssSelector	none	Send only this part of the page.
Remove CSS selector	removeCssSelector	none	Drop these elements first.
Render JavaScript pages in a browser when needed	useBrowserFallback	true
Max pages	maxPages	100
Max concurrency	maxConcurrency	5	Lower it if your LLM account hits rate limits.
Extra domains to skip	extraBlockedDomains	none	Added to the built-in list below.

Output

One dataset row per input URL, including the ones that failed or were skipped, so you can see what happened to every URL. The Overview view shows status, data and errors; Failed or skipped pages shows only the diagnostic columns.

This row is from a demo-mode run of the default input on 2026-10-21 (link list shortened to three entries):

{
  "url": "https://quotes.toscrape.com/",
  "loadedUrl": "https://quotes.toscrape.com/",
  "status": "success",
  "mode": "demo",
  "fetchedWith": "http",
  "httpStatus": 200,
  "data": {
    "title": "Quotes to Scrape",
    "metaDescription": null,
    "canonicalUrl": null,
    "language": "en",
    "h1": ["Quotes to Scrape"],
    "h2": ["Top Ten tags"],
    "wordCount": 278,
    "linkCount": 49,
    "links": [
      { "text": "Quotes to Scrape", "url": "https://quotes.toscrape.com/" },
      { "text": "Login", "url": "https://quotes.toscrape.com/login" },
      { "text": "(about)", "url": "https://quotes.toscrape.com/author/Albert-Einstein" }
    ]
  },
  "validationErrors": [],
  "error": null,
  "attempts": 0,
  "contentChars": null,
  "truncated": false,
  "provider": null,
  "model": null,
  "usage": null,
  "note": "Demo mode: no LLM API key was given, so this row holds page facts read directly from the HTML (title, meta, headings, links). Add an API key to extract your own fields.",
  "extractedAt": "2026-10-21T05:09:22.918Z"
}

In LLM mode the row has the same keys, without note. data holds your fields, and provider, model, attempts, contentChars and usage are filled in.

Key	Meaning
url	URL as given in the input
loadedUrl	Final URL after redirects
status	See the table below
mode	llm or demo
fetchedWith	http or browser
httpStatus	Status code of the page response
data	The extracted object or array, or the demo shape
validationErrors	Schema errors left after the retry, e.g. /price must be number
error	Reason for a skip or failure
attempts	LLM calls made for the page: 1, or 2 after a retry
contentChars	Length of the markdown sent to the model
truncated	true if the page was cut at maxContentChars
provider, model	Which LLM produced the row
usage	inputTokens and outputTokens billed by your provider for this page, retries included
note	Demo mode only
extractedAt	ISO timestamp

status	Charged	What happened
success	yes, in LLM mode	data matches the schema
validation_failed	no	Two answers in a row failed the schema; the last one is in data
llm_error	no	Provider error, refusal, or output cut off. A rejected key or unknown model stops the run.
fetch_failed	no	HTTP error, timeout, non-HTML content, or page over 8 MB
no_content	no	Nothing readable left after cleanup
blocked_domain	no	On the blocked list, or redirected there
not_allowed	no	The URL, or a redirect, points to a private or local address (localhost, internal networks, cloud metadata)
robots_disallowed	no	The site's robots.txt disallows it
invalid_url	no	Not an http(s) URL
skipped	no	The run stopped first (your cost limit was reached, or the key was rejected)

A SUMMARY record in the run's key-value store has the count of each status.

How much it costs

There are two separate costs.

• This Actor uses pay-per-event pricing: about $0.01 per page-extracted event plus Apify's small per-run start fee. The Pricing tab shows the current figures. A page is charged only when its result passed schema validation. Demo runs, failures and skipped pages are never charged per page.
• Your LLM provider bills your own account for tokens. The usage field shows the exact numbers per page. As a rough guide, at about 4 characters per token, a 20,000-character page is about 5,000 input tokens. With claude-haiku-5-5 at $0.10 per million input tokens and $0.50 per million output tokens (Anthropic's list price on 2026-10-21), that is about $0.0005 of input plus the reply. Larger models cost more.

If you set a maximum cost for the run, the Actor checks the remaining budget before each LLM call. Once the budget is used up, the remaining pages are saved as skipped without being sent to your provider.

Blocked sites and robots.txt

These are always skipped because their terms forbid automated collection and their owners enforce that, in some cases in court: linkedin.com, facebook.com, instagram.com, threads.net, amazon (all country domains), x.com, twitter.com, youtube.com, tiktok.com, and Google search result pages (google.*/search). Other Google pages, such as developer docs, are not blocked. You can add more hosts with Extra domains to skip, but you cannot remove these.

Only public addresses are fetched. URLs that point to localhost, private networks, link-local or cloud metadata addresses, or hostnames that resolve to them, are saved with status not_allowed and never requested. Every redirect, and every request the headless browser makes, is checked the same way.

Every URL is checked against its site's robots.txt for the user agent AIStructuredExtractor. Disallowed URLs are saved with status robots_disallowed and not fetched. If a site's robots.txt returns a server error or times out, the site is treated as disallowed.

The Actor does not log in, solve captchas, rotate proxies or change its user agent to get past a block. Requests use one identifiable user agent: Mozilla/5.0 (compatible; AIStructuredExtractor/0.1; +https://apify.com/store).

Your API key

The key field is an Apify secret input, so Apify stores it encrypted along with the rest of the run input. The Actor sends it only to the provider you chose. It never logs the key or writes it to the dataset or to any record of its own, and it strips the key from provider error messages before they are logged.

Limits

• Pages behind a login, a paywall or bot protection fail with fetch_failed. There is no workaround on purpose.
• JavaScript detection is a heuristic. A page that ships some server-rendered text but loads the data you want later by script may not trigger the browser. Use a narrower Content CSS selector and check contentChars if results look empty.
• The browser fallback waits for the network to go quiet for up to 15 seconds. Infinite scroll, "load more" buttons and content behind clicks are not handled.
• Only HTML and plain-text pages are read. PDFs, images and other files fail with fetch_failed.
• Long pages are cut at maxContentChars. Items past the cut are missing from list pages, and the row says truncated: true.
• The model can still misread a page. Schema validation catches wrong types, missing keys and extra keys, but it cannot tell whether a correctly typed value is the right one. Spot-check results before relying on them.
• Accuracy and cost depend on the model you choose. Small, cheap models are more likely to skip items on long lists, mix up rows in dense tables, or fill a field from the wrong part of the page. If that happens, try a larger model through Model.
• One URL gives one row. The Actor does not follow links or crawl a site.

Running it locally

npm install
npx playwright install chromium
npx apify-cli run

With no key it runs in demo mode. To test LLM mode locally without writing the key into storage/key_value_stores/default/INPUT.json, export it as LLM_API_KEY in your shell before npx apify-cli run. This environment fallback is ignored on the Apify platform.

npm test runs the unit tests. test/mock-llm.mjs is a local stand-in for the Anthropic and OpenAI APIs, used to test retries, charging and error handling without spending tokens.

Feedback

Open an issue on the Issues tab with the URL, your schema and the run ID. Please leave your API key out.