AI Web Scraper: any public page to validated JSON

Paste a few URLs, describe in plain English what you want from each page, and get back one clean JSON row per page. Add a JSON Schema if you want exact field names and types. Every row is checked against your schema before it is saved, and you only pay for rows that pass.

What it does

• Reads public web pages. No login, no paywalls, no CAPTCHA solving.
• Turns each page into clean text (menus, scripts, footers and cookie banners removed).
• Asks a language model to pull exactly what you described, as JSON.
• Validates the answer against your schema. If it does not match, the model gets one retry with the error. If it still fails, the row is marked extraction_failed and you are not charged for it.
• Opens the page in a headless browser only when plain HTTP returns almost no text, so simple pages stay fast and cheap.
• Respects robots.txt and runs at low concurrency (3 pages at a time).

How to use it

1. Add your Start URLs. Each URL becomes one row.
2. Write your Extraction request, for example: "List each pricing plan with its name and monthly price in USD."
3. Optional: paste an Output schema (JSON Schema). Without one, the model picks short snake_case keys.
4. Set a Page limit and run.

Input

Field	Required	What it is
Start URLs	yes	Public pages to read
Extraction request	yes	Plain English description of the data you want
Output schema	no	JSON Schema the data field must match
Fields	no	List of fields to extract: name, type, description. See "Field lists"
JSON Schema	no	Raw JSON Schema instead of a field list. Use one of the two, not both
Page limit	no	Max URLs per run (default 10)
Model	no	Any OpenRouter model ID (default google/gemini-2.5-flash-lite)
Browser fallback	no	Render JavaScript pages when plain HTTP is empty (default on)
Personal data	no	Off by default: emails and phone numbers are replaced with [redacted]
OpenRouter API key	no	Use your own key instead of paying tokens through Apify

Example input, the same one the form is filled in with. It reads one pricing page and returns the plan names and monthly prices.

{
  "startUrls": [{ "url": "https://apify.com/pricing" }],
  "instructions": "List each pricing plan with its name and monthly price in USD.",
  "outputSchema": {
    "type": "object",
    "properties": {
      "plans": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "name": { "type": "string" },
            "monthlyPriceUsd": { "type": ["number", "null"] }
          },
          "required": ["name"]
        }
      }
    },
    "required": ["plans"]
  },
  "maxPages": 3
}

You can leave out outputSchema. Then the model picks short snake_case keys for the data:

{
  "startUrls": [{ "url": "https://apify.com/pricing" }],
  "instructions": "List each pricing plan with its name and monthly price in USD."
}

Without a field list, startUrls and instructions are required. maxPages defaults to 10.

Output

One row per URL. The data field follows your schema; the values below show the shape of a row, not a measured result:

{
  "url": "https://apify.com/pricing",
  "status": "ok",
  "data": { "plans": [ { "name": "Starter", "monthlyPriceUsd": 39 } ] },
  "error": null,
  "title": "Pricing",
  "fetchedVia": "http",
  "scrapedAt": "2026-10-01T09:00:00+00:00"
}

status is one of ok, fetch_failed, blocked_by_robots, blocked, empty_page, extraction_failed, schema_invalid, invalid_input. Only ok rows are charged.

Cost example

You pay two things, both through your Apify account:

1. Page extracted: $0.004 per successful row, which is $4 per 1,000 pages. Failed rows are free.
2. Model tokens at OpenRouter's own price, billed by Apify's official OpenRouter Actor. With the default model and a typical page (about 3,000 input and 300 output tokens) that is roughly $0.0004 per page, or about $0.40 per 1,000 pages. Apify's free plan pays a higher token rate, so a paid plan is better for large runs.

So 1,000 pages cost about $4.40 in total. You can see the token count for the run in the final status message.

The page price is lower on paid plans: $0.0035 per page on Starter, $0.003 on Scale and $0.0025 on Business. The prefilled run reads 3 pages, so it costs 3 x $0.004 = $0.012 on the Free plan, plus the model tokens and a start event of $0.00005.

Field lists

Instead of writing a JSON Schema, give a list of fields. Each field has a name, a type and an optional description. The type is one of string, number, boolean, url, date, array. Every output row then has exactly these keys.

Before a row is saved, the Actor converts simple values and checks the result:

• Numbers: "$1,299" becomes 1299, "£51.77" becomes 51.77.
• Booleans: "Yes" becomes true.
• Dates: "March 5, 2024" becomes "2024-03-05".
• Arrays: "a, b, c" becomes ["a", "b", "c"].
• A value that is not on the page is null. Keys you did not ask for are dropped.

If a row still does not match, the model gets one retry with the list of problems. If it still fails, the row gets the status schema_invalid, the problems field lists what is wrong, and you are not charged. Rows that pass keep the normal ok status and price. A wrong field definition (for example an unknown type) ends the run with one free invalid_input row.

If you need nested objects or limits such as minimum, use jsonSchema instead. It takes a JSON Schema with "type": "object", and the same checks and statuses apply. If you set fields or jsonSchema, outputSchema is ignored, and "Extraction request" becomes optional.

Example input:

{
  "startUrls": [
    { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" },
    { "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html" }
  ],
  "fields": [
    { "name": "title", "type": "string", "description": "Book title" },
    { "name": "price", "type": "number", "description": "Price in GBP" },
    { "name": "in_stock", "type": "boolean" },
    { "name": "copies_available", "type": "number" },
    { "name": "page_url", "type": "url" }
  ]
}

Rows from a local test run on those two pages. The page fetching and the checks were real. The model reply was replaced by a fixed test script, so this shows the conversion and the check, not model quality. The script answered with text such as "£51.77" and "Yes":

{ "title": "A Light in the Attic", "price": 51.77, "in_stock": true, "copies_available": 22, "page_url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" }
{ "title": "Tipping the Velvet", "price": 53.74, "in_stock": true, "copies_available": 20, "page_url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html" }

A row the check rejects looks like this (test run, price returned as the word "free"):

{ "status": "schema_invalid", "error": "price: 'free' is not of type 'number', 'null'", "problems": ["price: 'free' is not of type 'number', 'null'"] }

Bulk lists

Use a long list of page addresses from an Apify Dataset or from a file. Pick the Dataset under Dataset with the list and write the name of its field under Field in the dataset. Leave the field name empty to use the first text field. Or paste the web address of a .txt, .csv or .json file under List file URL. A .csv file uses its first column. A header row is skipped. The file can be up to 10 MB. Entries from the Dataset and the file are added to the ones typed under Start URLs, and duplicates are removed. The price per extracted page and the way LLM tokens are billed do not change. The page limit still applies to the merged list.

Related Actors

• Web Page to Markdown for AI: Clean Markdown from any web page.
• Article Extractor: Article text and metadata from URLs.
• RSS Feed Reader: Items from RSS and Atom feeds.
• Sitemap Extractor: All URLs from a sitemap.
• PDF Extractor API: Text from PDF files.

FAQ

Is web scraping legal? Reading publicly available pages is generally lawful, but you are responsible for how you use the data. Respect each site's terms, copyright and privacy law (for example GDPR). This Actor only reads public pages, follows robots.txt, does not bypass anti-bot challenges and redacts emails and phone numbers by default. If you are unsure, ask a lawyer.

Why did a page come back as blocked? The site showed an anti-bot challenge. We do not try to get around it. The row is free.

Can it crawl a whole site? Not in this version. It reads exactly the URLs you give it. That keeps runs predictable and cheap.

Can I use my own model key? Yes. Put your OpenRouter key in the secret field and tokens are billed to that key instead.

What if the model makes something up? The prompt tells it to use only facts on the page and to return null for missing fields, and your schema catches wrong types. Keep schemas specific for best results.

Can it read a list of items, like products or books? Yes. On a book store page I asked for the first 5 books with their title and price, and all 5 came back with the right prices. Name the number of items you want in the instruction, and keep the schema to fields that appear as text.

Why is a field empty when I can see it on the page? The Actor reads the text of the page. A value that is only drawn with an icon or a style, such as a star rating, is not in the text and comes back as null. In the same book store test, the star ratings came back empty for this reason.