Turn web pages, or another Actor's dataset, into structured JSON with an LLM: an AI web scraper for pages you already have. Give a JSON schema or plain-English fields; every answer is validated, and you pay $5 per 1,000 valid pages plus the model's tokens. No browser.
Versions follow MAJOR.MINOR.PATCH (src/version.py); Apify shows MAJOR.MINOR from .actor/actor.json.
Every run logs its version and records it in the RUN_STATS key-value record.
About future changes in output: what a model reads off a page can differ between models, and between runs
of the same model. Every answer is checked against your schema, and one that doesn't match comes back with
valid: false and the reasons, uncharged.
1.0.7 (2026-10-16)
A run that returns nothing (every input bad, private, blocked or unreachable, or an empty input) now ends
SUCCEEDED with an empty dataset and the reason as its status message and in the log, instead of FAILED. Only an
unexpected error still fails the run. RUN_STATS keeps the reason as before.
No change to output fields or price.
1.0.6 (unreleased)
Fixed: one flagged page no longer stops the run. When the model provider's moderation flags a page's text,
OpenRouter answers with HTTP 403, the same status as a refused token, and the run stopped there and failed. That
page now gets a row with the reason ("the model provider's moderation flagged this input (...)"), uncharged, and
the other pages carry on. A refused token or account still stops the run at once.
A model that declines to answer now says so, with its reason ("the model declined to answer: ..."), instead of
"the model's answer isn't JSON: (empty)". Still a row with valid: false, uncharged.
1.0.5 (unreleased)
No change in what it does or charges. The model call, schema checks and validation now live in a shared module
(mms_llm) used by this actor and by AI for Every Row, which runs the same validated call on dataset rows
instead of web pages.
1.0.4 (2026-10-01)
A site that asks for a long wait (over 30 seconds) again after it was waited out is reported at once instead of
waited out a second time: one long wait per site per run. Two Shopify shops did that to the 1.0.3 run on Apify.
1.0.3 (2026-10-01)
Fixed: no charge for answers with nothing in them. An answer that matches your schema but holds no value
(every field null, blank or "N/A"-style, or only empty lists) now comes back valid: false with the reason,
uncharged (pagesEmpty in RUN_STATS). One real value, false and 0 included, still makes a valid page.
Stricter check before the model: bot-check pages served with HTTP 200 (botChecks), and thin pages that a
script fills in (under 15 words outside the menus and footer plus scripts, or the data waiting in an attribute
for a script, with no JSON-LD to read instead) are reported as such, not sent. Catches the three pages of the
2026-10-01 test that 1.0.2 charged for nothing; every page it read correctly is still read.
Waits out longer Retry-After answers: up to 90 seconds (was 30), once per site, at most 3 minutes of such
long waits per run. (Two Shopify shops asked Apify's servers to wait 60 seconds and asked again after the wait;
those pages are still reported with the reason, uncharged.)
Narrower AI opt-out rule: only the general AI tokens count (Google-Extended, Applebot-Extended, CCBot and
OpenAI's, Anthropic's and Perplexity's crawlers and user fetchers). A site that blocks only one company's crawler
(Bytespider, Meta-ExternalAgent, Amazonbot, ...) is read now, e.g. GitHub and Notion.
1.0.2 (2026-10-01)
Fixed: an answer longer than 2,048 tokens said it was "cut off at 4096 tokens". Apify's OpenRouter proxy
returns at most 2,048 answer tokens, so the actor now asks for 2,048 and the error names that limit and what to
do (fewer fields, shorter values, fewer items per page). Such pages were and stay uncharged.
1.0.1 (2026-10-01)
Fixed: pages built from web components with declarative shadow DOM (MDN's docs, for one) were reported as
"needs JavaScript" although their text is in the HTML. Their content is read now; with the default fullPage
setting, text inside a <template shadowrootmode> goes to the model too, as a browser would show it.
Store title is now "AI Data Extractor & AI Web Scraper: Pages to Structured JSON"; the README compares it with
browser-based AI web scrapers and says which pages it can't read. No change to input, output or price.
1.0.0 (2026-09-30)
First release: web pages, or the pages in another actor's dataset, to JSON that matches your schema.
Input:urls, or datasetId (+ optional datasetUrlField) to read page URLs from any dataset and chain
after another actor with {{resource.defaultDatasetId}} (the shared reader of page-to-markdown and seven more
actors: url, pageUrl, link, website, loadedUrl found automatically; Google Maps links never used;
tracking parameters dropped; each row carries sourceTitle, sourcePlaceId, sourceIndex).
What to extract:fields in plain English (each becomes a camelCase key; a hint after a colon goes to the
model) or a JSON Schema in schema (overrides fields), plus optional instructions.
Reading pages: the shared guarded client (robots.txt, AI-crawler opt-outs, private-network guard, ports 80
and 443, HTML only, 5 MB cap), no browser: pages that need JavaScript are reported, not sent to the model.
pageContentfullPage (default) keeps headers, footers and sidebars, minus scripts, forms, hidden elements and
navigation menus; mainContent sends only the main content. The page's schema.org JSON-LD goes along.
maxTokensPerPage (default 8,000) is a hard cap: a longer page keeps its start and end, truncated: true.
The model: one call per page through Apify's OpenRouter proxy with the run's own token, so tokens are billed
to the account running the actor; model takes any OpenRouter id (default openai/gpt-4.1-mini). The schema
goes as a structured-output format, strict when it qualifies; a provider that refuses it in strict mode gets it
again as a hint. Every answer is validated here: valid, errors (at most 5), data kept for inspection.
A proxy that refuses the run (401/402/403) stops it at once, with the reason.
Charging: rows are free; the custom page-extracted event is charged once per page whose answer validated,
after the row is in the dataset. Failed, blocked and invalid pages are rows with the reason, never charged.
maxPages and the maximum cost per run are honoured before any page is fetched: a slot is claimed per page and
given back when the page doesn't validate.
Failure isolation: one page (or one bug on it) never affects another. A run where no page validated fails with
the first reason; the rows stay in the dataset.
Sources and terms (ACTOR-CHECKLIST section 0): the pages are the user's own URLs, read like page-to-markdown
reads them, AI opt-outs honoured. The model endpoint is Apify's own, documented for actors
(https://docs.apify.com/sdk/python/docs/guides/ai-agents , read 2026-09-29: "The token usage is billed against the
Apify account running the Actor, so no provider API key is required"; "The Actor authenticates with the proxy
using the APIFY_TOKEN that the platform injects into every run"). No credential or identity of ours is used.