Schema Extractor

Turn a list of URLs plus a JSON Schema into validated, typed records. It replaces Apify's ai-web-scraper (~$0.03/page) with a much cheaper, more predictable pipeline: fetch -> strip the page down to its readable core -> ask an LLM to fill in your schema -> validate the result with jsonschema -> retry once on failure -> emit one row per record.

How it works

1. Fetch each page with httpx (no headless browser, no JS rendering).
2. Reduce the HTML: drop <script>/<style>/<nav>/<footer>/ads/menus, keep the <main>/<article>/largest-text-block content as headings, paragraphs, list items and table rows, capped at ~12,000 characters. Any application/ld+json blocks are kept verbatim, uncapped, since structured data is often the highest-signal part of a page.
3. Extract: the reduced content plus your JSON Schema is sent to the LLM, asking for JSON only.
4. Validate: the response is parsed and checked against your schema with jsonschema. On failure, one retry is made with the validation error appended to the prompt. If that also fails, the page becomes a single error row instead of crashing the run.
5. Emit: each row carries _confidence, _source, _model, _attempts alongside your schema's fields.

Input

Field	Type	Description
startUrls	array of {url}	Pages to start from.
schema	object (JSON Schema)	Shape of the record you want back from each page.
maxPages	integer, default 20, max 1000	Total pages to fetch (start URLs + discovered links).
followLinksMatching	string (regex), optional	When set, same-site links on each page matching this regex are crawled too (BFS), up to maxPages.
llmProvider	anthropic | openai | bedrock | none	Which LLM backend to use. none is a dry run (always returns {}), useful for testing a schema/crawl without spending tokens.
model	string, optional	Overrides the provider's default model.
apiKey	string (secret), optional	Bring your own key. If omitted, the Actor's bundled key is used where available (billed at the bundled price).
awsRegion, awsAccessKeyId, awsSecretAccessKey	string, optional	Bedrock credentials; fall back to AWS_REGION / AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars.
extractionInstructions	string, optional	Free-text guidance appended to the prompt (e.g. "prices are in EUR").
multipleRecordsPerPage	boolean, default false	If true, the LLM is asked for a JSON array and every element becomes its own output row.
maxConcurrency	integer, default 3, max 5	Pages fetched/extracted in parallel.

Output example

Given the schema:

{
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "price": { "type": "number" },
    "currency": { "type": "string" },
    "inStock": { "type": "boolean" }
  },
  "required": ["title", "price"]
}

a product page produces a dataset row like:

{
  "title": "Widget Pro 3000",
  "price": 49.99,
  "currency": "USD",
  "inStock": true,
  "_confidence": 1.0,
  "_source": "https://shop.example.com/widget-pro-3000",
  "_model": "claude-haiku-4-5",
  "_attempts": 1,
  "_extracted_at": "2026-08-29T12:00:00+00:00"
}

A job-listing page schema for multipleRecordsPerPage: true:

{
  "type": "object",
  "properties": {
    "role": { "type": "string" },
    "location": { "type": "string" },
    "remote": { "type": "boolean" }
  },
  "required": ["role"]
}

produces one row per job posting found on the page, each with its own _confidence / _source / _model / _attempts.

A page the LLM couldn't fit to the schema, even after one retry, produces a single row like:

{
  "error": "'price' is a required property",
  "_source": "https://shop.example.com/broken-page",
  "_model": "claude-haiku-4-5",
  "_attempts": 2,
  "_confidence": 0.0
}

Pricing events

This Actor uses Apify's pay-per-event pricing:

Event	Price	When
page-extracted-byok	$0.004 / page	You supplied your own apiKey.
page-extracted-bundled	$0.01 / page	No apiKey supplied; the Actor's bundled key/credits are used.

One event is charged per successfully extracted row (a page with multipleRecordsPerPage produces several rows, each charged). You are never charged for a row with no data in it -- a page that couldn't be fetched, an LLM response that never validated against your schema after retrying, or a legitimate "no records matched this schema" result -- even though an LLM call may have been attempted; you only pay for data you actually received. These rows still appear in your output dataset (never silently dropped, each with a clear explanation), just without a charge attached.

MCP tool

This Actor is usable as an MCP tool:

extract(url: string, schema: object) -> object[]

It runs a single-page extraction (no crawling) against url with the given JSON Schema and returns the validated record(s).

Limitations

• No JavaScript rendering. Pages that build their content client-side (SPA shells with an empty initial HTML payload) will not have that content to extract - only application/ld+json and whatever's in the static HTML.
• Token limits. Page content is capped at ~12,000 characters before it reaches the LLM; extremely long pages will lose tail content (JSON-LD is kept regardless of the cap).
• One retry only. If the LLM can't produce schema-valid JSON in two attempts, the page is recorded as an error row rather than retried further.
• followLinksMatching crawls same-site links only, breadth-first, and respects robots.txt and a minimum 0.5s per-host delay - it is not meant as a general-purpose crawler.