AI Web Scraper: JSON Schema Extractor

AI web scraper that turns page URLs into JSON that is validated against your JSON Schema. Give it URLs, a schema (or just a list of field names) and optional instructions. It downloads each page over plain HTTP, cleans it into compact Markdown, asks an LLM for structured output, validates the answer with a real JSON Schema validator (ajv), and retries with the validation errors fed back to the model. Pages that cannot be extracted or validated are never charged.

This AI scraper is built for people who want predictable, typed data from messy pages (products, listings, articles, job posts, company pages, documentation) without writing one parser per site. It works with your own OpenAI, OpenRouter, Anthropic or Gemini key (BYOK), so the LLM tokens are billed directly by your provider at their price.

What does AI Web Scraper: JSON Schema Extractor do?

• Takes a list of page URLs and a JSON Schema (draft-07 style) or a plain field list like ["title", "price: number", "tags: string[]"].
• Fetches each page with plain HTTP (no browser), strips scripts, styles, navigation and footers, keeps headings, tables, lists, absolute links and JSON-LD structured data, and converts it to Markdown.
• Sends the Markdown plus your schema to an LLM and asks for one JSON object.
• Validates the answer with ajv. If it does not match, the exact validation errors are sent back to the model and it tries again (default 2 retries, up to 3).
• Pushes one dataset item per page with status, data, token counts and the model used.
• Optional list-page to detail-page crawl: give a listing URL, turn on "Follow links", and the Actor finds the detail links and extracts each one, with hard limits (maxDetailPagesPerListing, maxPages, maxItems).
• Pages that render their content only with JavaScript are reported clearly as needs-javascript (no tokens spent, no charge). There is no browser fallback in this version.

Why use AI Web Scraper: JSON Schema Extractor?

1. Strict schema, not just a prompt. Output is validated with ajv and retried with the errors. You get the shape you asked for, or a clearly marked failure.
2. You pay only for pages that work. page-passthrough is billed only after a page is extracted and validated. Fetch errors, blocked pages, JavaScript-only pages and validation failures cost $0 in Actor fees.
3. HTTP-first is much cheaper. No browser means a page costs a fraction of the compute of a headless Chrome session. With your own key the Actor fee is $3 per 1,000 pages (FREE plan price, lower on paid plans), plus tokens at your provider's price. Check the current price of other AI scraping Actors on the Store and compare the per-page cost.
4. Bring your own key and model. OpenAI-compatible endpoints (OpenAI, OpenRouter with a custom base URL, Groq, Together, Mistral, self-hosted), Anthropic and Google Gemini. Model tiers: a cheap fast model by default and an accurate model when you need it.
5. Built for agents and automation. Array input, JSON in and JSON out, a saved OUTPUT summary and a stable dataset format make it easy to call from MCP clients, n8n, Make, Zapier and plain HTTP.

How much does it cost to scrape with an AI web scraper?

Two events exist. Use page-passthrough (the default) with your own API key. page-inclusive is an optional all-inclusive mode for which no key is needed; it only works if the Actor owner has enabled it.

Event	What is billed	FREE	BRONZE	SILVER	GOLD
page-passthrough	One page extracted and validated (LLM tokens billed by your provider)	$0.0030	$0.0027	$0.0024	$0.0021
page-inclusive	One page extracted and validated, LLM cost included (fast tier)	$0.0080	$0.0072	$0.0064	$0.0056

Apify's standard run start event applies as usual. No charge for failed pages.

Example: 1,000 pages with your own key on the FREE plan = 1,000 x $0.003 = $3.00 in Actor fees, plus your provider's tokens. As a rough guide (approximate public list prices, check your provider), a typical product page is about 3,000 to 8,000 input tokens and 300 to 600 output tokens, which costs around $1 per 1,000 pages on a Gemini Flash-Lite class model and a few dollars per 1,000 pages on a GPT-4.1-mini class model. Accurate-tier models cost several times more. Pages are truncated to 60,000 characters by default (maxInputChars) so one huge page cannot create a surprise bill.

Cost limits: set Apify's Maximum cost per run (maxTotalChargeUsd) to cap the Actor fee. Also use maxItems and maxPages. For LLM tokens, set a spend limit with your provider as well, because those are billed outside Apify.

Free plan: free Apify plan users get up to 25 results per run, then the Actor stops normally with the message "Free tier limit reached (25 results). Upgrade to a paid Apify plan for unlimited results."

Input

Field	Type	Description
urls	array of strings	Page URLs to extract. Main input.
schema	object	JSON Schema of the object to extract (type object at the top).
fields	array of strings	Simple alternative to schema: name, name: type, optional - description. Types: string, number, integer, boolean, string[], number[].
instructions	string	Optional hints, for example "Prices are in EUR".
llmProvider	select	openai-compatible (default), anthropic, gemini.
apiKey	secret string	Your own LLM API key. Empty means preview mode (no LLM call, no charge).
baseUrl	string	For OpenAI-compatible APIs, for example https://openrouter.ai/api/v1.
modelTier	select	fast (default) or accurate.
model	string	Exact model name override. Required with a custom base URL.
billingMode	select	byok (default) or inclusive (only if enabled by the owner).
maxRetries	integer	Validation retries 0 to 3 (default 2).
maxItems	integer	Stop after this many extracted pages (default 20).
followLinks	boolean	Treat URLs as listing pages and extract their detail pages.
detailLinkSelector	string	CSS selector of detail links (optional).
detailUrlPattern	string	Regex that detail URLs must match (optional).
maxDetailPagesPerListing	integer	Hard limit per listing page (default 10).
maxPages	integer	Hard limit of all fetches, including listings and failures (default 50).
contentSelector	string	Restrict the content sent to the model to a part of the page.
includeImages	boolean	Keep image URLs (auto-enabled when the schema mentions images).
includeMarkdown	boolean	Add the cleaned Markdown to every item.
maxInputChars	integer	Max characters of page content sent to the model (default 60,000).
maxConcurrency	integer	Pages processed in parallel (default 3).
requestTimeoutSecs	integer	Page download timeout (default 30).
proxyConfiguration	object	Apify proxy settings (default: datacenter proxy).
useResidentialFallback	boolean	Retry blocked pages through residential proxy (needs access on your plan).

Default behavior of the prefilled run: it fetches one public demo page and, because no API key is given, returns a preview (the Markdown that would be sent to the model) without charging anything.

Example 1: schema with your own OpenAI key

{
  "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
  "schema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "price": { "type": ["number", "null"], "description": "Price as a number" },
      "in_stock": { "type": ["boolean", "null"] }
    },
    "required": ["title", "price"],
    "additionalProperties": false
  },
  "llmProvider": "openai-compatible",
  "apiKey": "sk-...",
  "modelTier": "fast"
}

Example 2: plain field list through OpenRouter

{
  "urls": ["https://example.com/jobs/123", "https://example.com/jobs/124"],
  "fields": ["job_title", "company", "salary_min: number", "salary_max: number", "remote: boolean", "skills: string[]"],
  "instructions": "Salaries are yearly in USD. Leave a number empty if not stated.",
  "llmProvider": "openai-compatible",
  "baseUrl": "https://openrouter.ai/api/v1",
  "apiKey": "sk-or-...",
  "model": "google/gemini-2.5-flash-lite"
}

Example 3: listing page to detail pages with hard limits

{
  "urls": ["https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"],
  "followLinks": true,
  "maxDetailPagesPerListing": 10,
  "maxPages": 15,
  "maxItems": 10,
  "fields": ["title", "price: number", "upc", "availability"],
  "llmProvider": "anthropic",
  "apiKey": "sk-ant-...",
  "modelTier": "fast"
}

Output example

This is an illustrative item (not from a live run):

{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "status": "success",
  "data": { "title": "A Light in the Attic", "price": 51.77, "in_stock": true },
  "error": null,
  "attempts": 1,
  "provider": "openai-compatible",
  "model": "gpt-4.1-mini",
  "inputTokens": 1480,
  "outputTokens": 41,
  "markdownChars": 3620,
  "truncated": false,
  "sourceListingUrl": null,
  "charged": "page-passthrough",
  "extractedAt": "2026-10-19T12:00:00.000Z"
}

status is one of: success, validation-failed, llm-error, needs-javascript, blocked, not-found, fetch-failed, unsupported-content-type, no-links-found, dry-run (preview mode) and error. Only success rows are charged. Failed rows are pushed for free so you can see why. A run summary (counts, charged events, tokens, warnings, stop reason) is saved in the key-value store under OUTPUT.

How to use the AI web scraper (tutorial)

1. Open the Actor on Apify and paste your page URLs into "Page URLs".
2. Describe the data. Paste a JSON Schema into "JSON Schema", or leave it empty and list fields in "Fields". Add short property descriptions, they guide the model.
3. Add your API key. Choose the provider, paste the key, and pick the "fast" tier. For OpenRouter pick OpenAI-compatible and set the base URL to https://openrouter.ai/api/v1.
4. Start with a small run. Keep maxItems at 5 and check the dataset. If rows fail validation, add instructions, tighten descriptions, set a contentSelector, or switch to the "accurate" tier.
5. Scale up. Raise maxItems, schedule the run, set Maximum cost per run, and export the dataset as JSON, CSV or Excel or fetch it by API.

Run via API

curl -X POST "https://api.apify.com/v2/acts/tiger19935~schema-ai-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
    "fields": ["title", "price: number"],
    "apiKey": "sk-...",
    "maxItems": 1
  }'

Node.js:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('tiger19935/schema-ai-extractor').call({
    urls: ['https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'],
    fields: ['title', 'price: number'],
    apiKey: process.env.OPENAI_API_KEY,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.filter((i) => i.status === 'success').map((i) => i.data));

MCP: the input is plain JSON with arrays and simple fields, so the Actor can be exposed as a tool through the Apify MCP server. An agent passes urls, a schema and your key, and receives validated JSON rows. Never paste an API key into a shared agent prompt, store it as a secret in your MCP client configuration instead.

Integrations

• n8n and Make: use the Apify module, run the Actor, then read dataset items. Filter on status = success and use the data object.
• Zapier: trigger the Actor from a schedule or a form, then send data to your app.
• Google Sheets: use the Apify integration or export CSV/Excel from the dataset.
• Webhooks: call a URL when a run finishes and fetch the dataset items.
• MCP and agents: see the section above.

Legal / ethical use

The Actor downloads publicly available pages without logging in and does not bypass authentication. You are responsible for how you use the data. Check the target site's terms of service and robots.txt, respect copyright and database rights, and do not collect personal data without a legal basis (for example under GDPR). Page content is sent to the LLM provider you choose, so make sure that is acceptable for the data involved, and review your provider's data handling terms. Do not use the Actor on pages that require authentication or that you are not permitted to process.

FAQ and support

Which sites work? Pages whose content is present in the raw HTML (server-rendered or static pages, many shops, blogs, directories, documentation). Pages that load content only with JavaScript return needs-javascript. Sites that block automated requests may return blocked; you can try the residential fallback if your plan has it. The Actor does not work on every site.

Do I need an API key? For extraction, yes (BYOK), unless the optional inclusive mode is enabled by the Actor owner. Without a key the Actor runs in preview mode and shows the Markdown that would be sent, with no charge.

What is charged when a page fails? Nothing in Actor fees. Your LLM provider may still bill tokens for attempts that did not validate, so keep maxRetries low when cost matters.

How accurate is it? It depends on the model and the page. Validation guarantees the shape, not the truth of every value. Use clear property descriptions, a contentSelector for noisy pages and the accurate tier for hard cases. Always spot check results.

Can it extract lists of items from one page? Yes. Define an array property in the schema (for example products with an object items definition). Very long pages are cut at maxInputChars, so split big lists across pages or raise the limit.

Can it follow pagination or crawl a whole site? No. It can follow links from a listing page to detail pages, up to maxDetailPagesPerListing per listing and maxPages overall. Pass each listing page URL yourself for more.

Which models are used by default? Fast tier: gpt-4.1-mini, claude-haiku-4-5, gemini-2.5-flash-lite. Accurate tier: gpt-4.1, claude-sonnet-4-5, gemini-2.5-pro. Override with the model field if your account uses different names.

Is my API key safe? It is stored as a secret input and is never written to logs or the dataset.

Need help or a feature? Open an issue in the Issues tab of this Actor. Replies come within 12 hours.

More tools by the same author

• Prozorro & TED Tenders Scraper - public tenders from Ukraine and the EU
• Billing & Payment Stack Detector - which payment gateway a website uses
• Website Technology Detector - tech stack of any site
• Decision-Maker Contacts - people and roles at a company

Related searches

ai web scraper, ai web scraper api, ai web scraper data, ai web scraper to csv, ai web scraper monitor, ai web scraper apify, ai web scraper mcp, ai scraper, ai scraper api, ai scraper data, json schema extraction, llm web scraper, structured data extraction with openai, openrouter scraper, schema based web scraping.