AI Web Scraper — Extract Website Data to Structured JSON

AI Web Scraper extracts data from the public web pages you list and returns it as structured JSON, using field names and plain-English descriptions you write instead of CSS selectors. It is built for analysts, developers and AI-agent builders who need clean, source-linked records from sites without a dedicated scraper, and it costs USD 20 per 1,000 pages processed on the FREE tier (USD 16 on GOLD), whether a page yields one row or fifty.

Free users: maximum 20 results per run.

What does the AI web scraper return?

Each dataset row holds the fields you defined plus three fixed fields:

Field	What it contains
your field names	One property per field you define, such as title, price, author or tags. null when the page does not contain the value.
source_url	The page the row was extracted from.
scraped_at	ISO 8601 timestamp of the extraction.
error	null on success; otherwise the reason, such as No items extracted or a model error. Error rows are never charged.

In list mode (the default) every repeating item on a page becomes its own row: product cards, search results, job listings, directory entries or quotes. With list mode off, each page returns one record, which suits product detail pages, articles and profiles. Export the dataset as JSON, CSV or Excel, or read it through the API.

How do I extract data from a website without CSS selectors?

Give the Actor a list of page URLs and a list of fields to pull from them; an AI model reads each page and fills in your fields.

1. Add one or more Start URLs. Only the pages you list are processed; links are not followed.
2. Define Fields to Extract as objects with name, a short description, and an optional type (number, array, boolean). Plain strings such as "title" also work.
3. Leave list mode on for pages with many repeating items, or turn it off for one record per page.
4. Bound the run with Max Items (default 100, up to 10,000 rows), Max Pages (default 20, up to 1,000 pages) and Max Content Characters (default 12,000, from 1,000 to 60,000 characters of page text per page).

The Actor downloads each page over plain HTTP through Apify's datacenter proxy (a proxyConfiguration input is ignored), strips scripts, styles and markup, and sends the first maxContentChars characters of text to the model with your field list. The model must answer in strict JSON with null for missing values, at temperature 0. Descriptions such as "price in USD as a number" or "ISO 8601 date" steer the format.

{
  "startUrls": [{ "url": "https://quotes.toscrape.com" }],
  "fields": [
    { "name": "text", "description": "the full quote text" },
    { "name": "author", "description": "name of the person who said the quote" },
    { "name": "tags", "description": "list of tag labels on the quote", "type": "array" }
  ],
  "listMode": true,
  "maxItems": 100,
  "maxCrawlPages": 20,
  "maxContentChars": 12000
}

If startUrls is empty, the demo page https://quotes.toscrape.com is used. A run on that page returned 10 rows in about 13 seconds on 2026-10-26.

When should I use it, and when not?

Use AI Web Scraper when you already have a list of public URLs, the layouts differ from site to site, and writing selectors for each one is not worth it. Typical jobs are product names and prices from long-tail shops, job titles and salaries from job boards, entries from public directories, and article titles, authors and dates for a content inventory or a RAG index.

Do not use it when:

• The page needs JavaScript, a login, cookies or a form. The Actor reads the raw HTML response without a browser, so content injected by JavaScript is not seen. Use a browser-based scraper instead.
• You need to crawl a whole site or follow pagination. List each page URL (for example /page/1/, /page/2/).
• You need full article bodies. Use Smart Article Extractor.
• You want HTML turned into JSON without a per-page AI cost. Use HTML to JSON Smart Parser.
• You need LinkedIn profiles. Use LinkedIn Profile Scraper.
• You need company or contact enrichment. Use AI Lead Enrichment.

How do I call it from the API?

JavaScript (apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('muhammadafzal/ai-web-extractor').call({
  startUrls: [{ url: 'https://quotes.toscrape.com' }],
  fields: [
    { name: 'text', description: 'the full quote text' },
    { name: 'author', description: 'name of the person who said the quote' },
  ],
  listMode: true,
  maxItems: 50,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python (apify-client):

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("muhammadafzal/ai-web-extractor").call(run_input={
    "startUrls": [{"url": "https://quotes.toscrape.com"}],
    "fields": [
        {"name": "text", "description": "the full quote text"},
        {"name": "author", "description": "name of the person who said the quote"},
    ],
    "listMode": True,
    "maxItems": 50,
})
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

For one-shot calls, POST https://api.apify.com/v2/acts/muhammadafzal~ai-web-extractor/run-sync-get-dataset-items returns the rows in the response; it waits up to 300 seconds, so start larger batches as normal runs. Schedules, webhooks, Make, Zapier and n8n work as with any Apify Actor.

Use it from an AI assistant

AI Web Scraper works from Claude, Cursor or any MCP client through Apify's MCP server. Add the single-tool URL https://mcp.apify.com/?tools=muhammadafzal/ai-web-extractor to your client, then ask, for example: "From https://books.toscrape.com, extract every book's title, price as a number and stock status, max 20 rows." The assistant writes the field list, starts the run and returns the rows. Name a row or page limit in the prompt, because each processed page is charged.

Output example

Two rows from the demo page, shortened:

[
  {
    "text": "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.",
    "author": "Albert Einstein",
    "tags": ["change", "deep-thoughts", "thinking", "world"],
    "source_url": "https://quotes.toscrape.com",
    "scraped_at": "2026-10-26T07:43:30.602Z",
    "error": null
  },
  {
    "text": "There are only two ways to live your life...",
    "author": "Albert Einstein",
    "tags": ["inspirational", "life", "live", "miracle", "miracles"],
    "source_url": "https://quotes.toscrape.com",
    "scraped_at": "2026-10-26T07:43:30.602Z",
    "error": null
  }
]

Pricing: how much does AI data extraction cost?

AI Web Scraper uses pay-per-event pricing, with platform usage and the AI model cost included. Free users: maximum 20 results per run.

Event	FREE	BRONZE	SILVER	GOLD	PLATINUM / DIAMOND
Page processed (a page that produced at least one saved row)	USD 0.02	USD 0.019	USD 0.019	USD 0.016	USD 0.016
Actor start (per started GB of allocated memory)	USD 0.005	USD 0.005	USD 0.005	USD 0.005	USD 0.005

These prices have applied since 17 September 2026, and no change is scheduled. The default 512 MB run (maximum 1,024 MB) pays one start event.

Worked totals on the FREE tier:

• The demo page (1 page, 10 rows): 1 × USD 0.02 + USD 0.005 = USD 0.025
• 100 listing pages: 100 × USD 0.02 + USD 0.005 = USD 2.005

Pages that fail to download, model errors, unparseable answers and pages with no items are written as error rows and are not charged. Set a maximum cost per run in Apify Console to cap spending.

How accurate and reliable is AI data extraction?

The model extracts only what is present in the page text it receives and returns null otherwise, but AI output is probabilistic, so validate critical fields. Other limits:

• Pages are processed one at a time. A failed download is retried up to two times; each page gets one model request with a 90-second timeout and is never sent to the model twice.
• The model's answer is limited to about 4,000 tokens per page in list mode and 600 in single mode. On very long lists only complete items are kept, so split long pages or request fewer fields.
• Only the first maxContentChars characters of page text are sent, so data further down a long page can be missed.
• If the AI service rejects the run's requests, the run stops early with a Blocked: status message and remaining pages are not processed or charged.

FAQ

Do I need my own AI or API key?

No. The AI model cost is paid by the Actor owner and included in the page price. You only need an Apify account.

Can I choose a different AI model?

No. Every run uses the same fixed model, which keeps prices predictable; a model value sent through the API is ignored.

Does it render JavaScript?

No. If the raw HTML does not contain the data, raising maxContentChars will not help; use a browser-based scraper for those pages.

Is there a free AI web scraper option?

Apify's free plan includes monthly platform credit you can spend here. On the free plan each run stops at 20 rows and 20 pages, and it also stops after the first page that fails or yields nothing.

Can it follow links or crawl a whole site?

No. It processes only the URLs in Start URLs, up to Max Pages per run. List paginated pages explicitly.

Why do results differ slightly between runs?

Pages change, and AI output can vary a little even at temperature 0. Clear field descriptions with explicit formats reduce the variance.

Which pages am I charged for?

Only pages that produced at least one saved row, once per page regardless of how many rows it yielded, plus the start event.

Related Actors

• Email Validator: check the syntax, domain and MX records of email addresses you extracted.
• LinkedIn Profile Scraper: public LinkedIn profiles from URLs, without cookies.
• Smart Article Extractor: full article text, headings and reading time.
• HTML to JSON Smart Parser: heuristic HTML-to-JSON conversion with an optional AI mode.
• AI Lead Enrichment: company and contact enrichment with ICP scoring.
• GEO Content Auditor: scores how well a page answers questions for AI search.

Legal and responsible use

Process only public content you are allowed to access, and respect website terms, copyright and data-protection laws such as GDPR and CCPA. Do not use the Actor to bypass access controls or to collect sensitive personal data. Page text is sent to an AI model for processing; treat extracted content as untrusted data. This is not legal advice.

Last verified: 2026-10-27