# AI Web Scraper: Any Page to Validated JSON, Plain English (`conserving_celerytop/ai-web-scraper`) Actor AI web scraper: give URLs and say in plain English what to pull, and get one schema-validated JSON row per page. $0.004 per page, failed pages are free. LLM tokens are billed at cost through your own Apify account. No login. - **URL**: https://apify.com/conserving_celerytop/ai-web-scraper.md - **Developed by:** [Don Mangu](https://apify.com/conserving_celerytop) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.50 / 1,000 page extractions This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## AI Web Scraper: any public page to validated JSON Paste a few URLs, describe in plain English what you want from each page, and get back one clean JSON row per page. Add a JSON Schema if you want exact field names and types. Every row is checked against your schema before it is saved, and you only pay for rows that pass. ### What it does - Reads public web pages. No login, no paywalls, no CAPTCHA solving. - Turns each page into clean text (menus, scripts, footers and cookie banners removed). - Asks a language model to pull exactly what you described, as JSON. - Validates the answer against your schema. If it does not match, the model gets one retry with the error. If it still fails, the row is marked `extraction_failed` and you are not charged for it. - Opens the page in a headless browser only when plain HTTP returns almost no text, so simple pages stay fast and cheap. - Respects robots.txt and runs at low concurrency (3 pages at a time). ### How to use it 1. Add your **Start URLs**. Each URL becomes one row. 2. Write your **Extraction request**, for example: "List each pricing plan with its name and monthly price in USD." 3. Optional: paste an **Output schema** (JSON Schema). Without one, the model picks short snake_case keys. 4. Set a **Page limit** and run. ### Input | Field | Required | What it is | |---|---|---| | Start URLs | yes | Public pages to read | | Extraction request | yes | Plain English description of the data you want | | Output schema | no | JSON Schema the `data` field must match | | Fields | no | List of fields to extract: name, type, description. See "Field lists" | | JSON Schema | no | Raw JSON Schema instead of a field list. Use one of the two, not both | | Page limit | no | Max URLs per run (default 10) | | Model | no | Any OpenRouter model ID (default `google/gemini-2.5-flash-lite`) | | Browser fallback | no | Render JavaScript pages when plain HTTP is empty (default on) | | Personal data | no | Off by default: emails and phone numbers are replaced with `[redacted]` | | OpenRouter API key | no | Use your own key instead of paying tokens through Apify | Example input, the same one the form is filled in with. It reads one pricing page and returns the plan names and monthly prices. ```json { "startUrls": [{ "url": "https://apify.com/pricing" }], "instructions": "List each pricing plan with its name and monthly price in USD.", "outputSchema": { "type": "object", "properties": { "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "monthlyPriceUsd": { "type": ["number", "null"] } }, "required": ["name"] } } }, "required": ["plans"] }, "maxPages": 3 } ``` You can leave out `outputSchema`. Then the model picks short snake_case keys for the data: ```json { "startUrls": [{ "url": "https://apify.com/pricing" }], "instructions": "List each pricing plan with its name and monthly price in USD." } ``` Without a field list, `startUrls` and `instructions` are required. `maxPages` defaults to 10. ### Output One row per URL. The `data` field follows your schema; the values below show the shape of a row, not a measured result: ```json { "url": "https://apify.com/pricing", "status": "ok", "data": { "plans": [ { "name": "Starter", "monthlyPriceUsd": 39 } ] }, "error": null, "title": "Pricing", "fetchedVia": "http", "scrapedAt": "2026-10-01T09:00:00+00:00" } ``` `status` is one of `ok`, `fetch_failed`, `blocked_by_robots`, `blocked`, `empty_page`, `extraction_failed`, `schema_invalid`, `invalid_input`. Only `ok` rows are charged. ### Cost example You pay two things, both through your Apify account: 1. **Page extracted**: $0.004 per successful row, which is $4 per 1,000 pages. Failed rows are free. 2. **Model tokens** at OpenRouter's own price, billed by Apify's official OpenRouter Actor. With the default model and a typical page (about 3,000 input and 300 output tokens) that is roughly $0.0004 per page, or about $0.40 per 1,000 pages. Apify's free plan pays a higher token rate, so a paid plan is better for large runs. So 1,000 pages cost about $4.40 in total. You can see the token count for the run in the final status message. The page price is lower on paid plans: $0.0035 per page on Starter, $0.003 on Scale and $0.0025 on Business. The prefilled run reads 3 pages, so it costs 3 x $0.004 = $0.012 on the Free plan, plus the model tokens and a start event of $0.00005. ### Field lists Instead of writing a JSON Schema, give a list of fields. Each field has a `name`, a `type` and an optional `description`. The type is one of `string`, `number`, `boolean`, `url`, `date`, `array`. Every output row then has exactly these keys. Before a row is saved, the Actor converts simple values and checks the result: - Numbers: `"$1,299"` becomes `1299`, `"£51.77"` becomes `51.77`. - Booleans: `"Yes"` becomes `true`. - Dates: `"March 5, 2024"` becomes `"2024-03-05"`. - Arrays: `"a, b, c"` becomes `["a", "b", "c"]`. - A value that is not on the page is `null`. Keys you did not ask for are dropped. If a row still does not match, the model gets one retry with the list of problems. If it still fails, the row gets the status `schema_invalid`, the `problems` field lists what is wrong, and you are not charged. Rows that pass keep the normal `ok` status and price. A wrong field definition (for example an unknown type) ends the run with one free `invalid_input` row. If you need nested objects or limits such as `minimum`, use `jsonSchema` instead. It takes a JSON Schema with `"type": "object"`, and the same checks and statuses apply. If you set `fields` or `jsonSchema`, `outputSchema` is ignored, and "Extraction request" becomes optional. Example input: ```json { "startUrls": [ { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" }, { "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html" } ], "fields": [ { "name": "title", "type": "string", "description": "Book title" }, { "name": "price", "type": "number", "description": "Price in GBP" }, { "name": "in_stock", "type": "boolean" }, { "name": "copies_available", "type": "number" }, { "name": "page_url", "type": "url" } ] } ``` Rows from a local test run on those two pages. The page fetching and the checks were real. The model reply was replaced by a fixed test script, so this shows the conversion and the check, not model quality. The script answered with text such as `"£51.77"` and `"Yes"`: ```json { "title": "A Light in the Attic", "price": 51.77, "in_stock": true, "copies_available": 22, "page_url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" } { "title": "Tipping the Velvet", "price": 53.74, "in_stock": true, "copies_available": 20, "page_url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html" } ``` A row the check rejects looks like this (test run, price returned as the word "free"): ```json { "status": "schema_invalid", "error": "price: 'free' is not of type 'number', 'null'", "problems": ["price: 'free' is not of type 'number', 'null'"] } ``` ### Bulk lists Use a long list of page addresses from an Apify Dataset or from a file. Pick the Dataset under **Dataset with the list** and write the name of its field under **Field in the dataset**. Leave the field name empty to use the first text field. Or paste the web address of a .txt, .csv or .json file under **List file URL**. A .csv file uses its first column. A header row is skipped. The file can be up to 10 MB. Entries from the Dataset and the file are added to the ones typed under **Start URLs**, and duplicates are removed. The price per extracted page and the way LLM tokens are billed do not change. The page limit still applies to the merged list. ### Related Actors - [Web Page to Markdown for AI](https://apify.com/conserving_celerytop/web-page-to-markdown): Clean Markdown from any web page. - [Article Extractor](https://apify.com/conserving_celerytop/article-extractor): Article text and metadata from URLs. - [RSS Feed Reader](https://apify.com/conserving_celerytop/rss-feed-reader): Items from RSS and Atom feeds. - [Sitemap Extractor](https://apify.com/conserving_celerytop/sitemap-url-extractor): All URLs from a sitemap. - [PDF Extractor API](https://apify.com/conserving_celerytop/pdf-text-extractor): Text from PDF files. ### FAQ **Is web scraping legal?** Reading publicly available pages is generally lawful, but you are responsible for how you use the data. Respect each site's terms, copyright and privacy law (for example GDPR). This Actor only reads public pages, follows robots.txt, does not bypass anti-bot challenges and redacts emails and phone numbers by default. If you are unsure, ask a lawyer. **Why did a page come back as `blocked`?** The site showed an anti-bot challenge. We do not try to get around it. The row is free. **Can it crawl a whole site?** Not in this version. It reads exactly the URLs you give it. That keeps runs predictable and cheap. **Can I use my own model key?** Yes. Put your OpenRouter key in the secret field and tokens are billed to that key instead. **What if the model makes something up?** The prompt tells it to use only facts on the page and to return null for missing fields, and your schema catches wrong types. Keep schemas specific for best results. **Can it read a list of items, like products or books?** Yes. On a book store page I asked for the first 5 books with their title and price, and all 5 came back with the right prices. Name the number of items you want in the instruction, and keep the schema to fields that appear as text. **Why is a field empty when I can see it on the page?** The Actor reads the text of the page. A value that is only drawn with an icon or a style, such as a star rating, is not in the text and comes back as null. In the same book store test, the star ratings came back empty for this reason. # Actor input Schema ## `startUrls` (type: `array`): Public pages to read. Each URL becomes one output row. No login pages. ## `listDatasetId` (type: `string`): Pick an Apify Dataset that holds your start URLs, one entry per item. They are added to the entries typed under Start URLs. ## `listDatasetField` (type: `string`): Name of the dataset field that holds the entry. Leave empty to use the first text field it finds. ## `listFileUrl` (type: `string`): Web address of a .txt, .csv or .json file with your start URLs, one entry per line. A header row is skipped. Up to 10 MB. ## `instructions` (type: `string`): Plain English: what should be pulled from each page. ## `fields` (type: `array`): Optional list of fields to extract. Each entry: name, type (string, number, boolean, url, date, array) and an optional description. Every row is checked against the list; simple values are converted ("$1,299" becomes 1299). Rows that fail the check get status schema_invalid and are free. If set, 'Extraction request' is optional. ## `jsonSchema` (type: `object`): Optional raw JSON Schema for the 'data' object. Use it instead of Fields, not together with it. Rows that fail the check get status schema_invalid and are free. ## `outputSchema` (type: `object`): Optional JSON Schema for the 'data' field. Rows that do not match it are retried once, then marked failed and not charged. ## `maxPages` (type: `integer`): Maximum number of URLs processed in this run. ## `model` (type: `string`): OpenRouter model ID. Token cost is billed to your Apify account at OpenRouter's price. ## `useBrowserFallback` (type: `boolean`): Open the page in headless Chromium when plain HTTP returns almost no text (JavaScript sites). ## `includePersonalData` (type: `boolean`): Off by default: emails and phone numbers in results are replaced with \[redacted]. Turn on only if you have a lawful basis. ## `openRouterApiKey` (type: `string`): Optional. Leave empty to pay tokens through your Apify account. If set, tokens are billed to this key instead. ## Actor input object example ```json { "startUrls": [ { "url": "https://apify.com/pricing" } ], "instructions": "List each pricing plan with its name and monthly price in USD.", "outputSchema": { "type": "object", "properties": { "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "monthlyPriceUsd": { "type": [ "number", "null" ] } }, "required": [ "name" ] } } }, "required": [ "plans" ] }, "maxPages": 3, "model": "google/gemini-2.5-flash-lite", "useBrowserFallback": true, "includePersonalData": false } ``` # Actor output Schema ## `results` (type: `string`): One row per page with url, status, data and error. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://apify.com/pricing" } ], "instructions": "List each pricing plan with its name and monthly price in USD.", "outputSchema": { "type": "object", "properties": { "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "monthlyPriceUsd": { "type": [ "number", "null" ] } }, "required": [ "name" ] } } }, "required": [ "plans" ] }, "maxPages": 3 }; // Run the Actor and wait for it to finish const run = await client.actor("conserving_celerytop/ai-web-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://apify.com/pricing" }], "instructions": "List each pricing plan with its name and monthly price in USD.", "outputSchema": { "type": "object", "properties": { "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "monthlyPriceUsd": { "type": [ "number", "null", ] }, }, "required": ["name"], }, } }, "required": ["plans"], }, "maxPages": 3, } # Run the Actor and wait for it to finish run = client.actor("conserving_celerytop/ai-web-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://apify.com/pricing" } ], "instructions": "List each pricing plan with its name and monthly price in USD.", "outputSchema": { "type": "object", "properties": { "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "monthlyPriceUsd": { "type": [ "number", "null" ] } }, "required": [ "name" ] } } }, "required": [ "plans" ] }, "maxPages": 3 }' | apify call conserving_celerytop/ai-web-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,conserving_celerytop/ai-web-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/2F7H7ExKvXpa4zcbb/builds/rE1GofCHT1iLOeqyN/openapi.json