# AI Web Scraper: Extract Structured Data from Websites (`zaiq/ai-web-scraper`) Actor

Describe the data you want in plain words (or list the fields) and get it from any web page as clean JSON rows: products, prices, listings, contacts, articles, tables. JavaScript pages too, crawl a site for the relevant pages, or just get each page as Markdown for your LLM.

- **URL**: https://apify.com/zaiq/ai-web-scraper.md
- **Developed by:** [Zaiq](https://apify.com/zaiq) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $6.40 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Web Scraper: any website to structured JSON with a prompt (products, prices, listings, contacts)

Describe the data you want **in plain words**, or list the fields, and get it from **any web page as clean JSON rows**: products and prices, listings, directories, contacts, articles, tables. No selectors, no code: the AI reads the page the way a person does.

Pages built by **JavaScript** open in a real Chrome when their HTML is empty. **Crawl mode** follows the links of a site that best match your prompt. With no prompt, every page comes back as **clean Markdown** for your own LLM.

### What you get

| Field | What it holds |
|---|---|
| Item rows | the fields your prompt asks for (or the ones you list), with `url`, `pageTitle` and `itemIndex`; numbers as numbers, prices with a separate `currency`, links as full URLs |
| One row per page (optional) | `items` (all items of the page) and `itemCount` |
| Markdown (optional) | `markdown`: the page's main content, for your own model |
| SUMMARY | pages and items per start URL, notes, the model used, pages opened in Chrome, what was charged |

### What people use it for

- **Product and price lists:** every product on a category page with price, rating and stock
- **Directories and listings:** companies, jobs, properties or events from any list page
- **Research tables:** the facts you name from articles, reports and documentation
- **Feeding an LLM:** pages as clean Markdown, without menus or cookie banners

### How to use it

1. Add one or more **start URLs**.
2. Write the **prompt**, e.g. *Every product: name, price, rating and link*, or list the **fields**.
3. Pick **single pages** or **crawl** (the most relevant pages of each site), then click **Start**.

### Input

| Field | Example | Notes |
|---|---|---|
| Start URLs | `https://www.example.com/products` | Pages to read (crawl mode starts there) |
| Prompt | `Every product: name, price, rating and link` | Empty: each page as Markdown |
| Fields | `name`, `price`, `rating` | Optional: the exact columns |
| Mode | `single` or `crawl` | Crawl follows the links that match the prompt |
| Most pages per site | `10` | Crawl mode |
| Model | `fast` | Gemini Flash; `smart` (GPT) and `claude` read harder pages |
| JavaScript pages | `auto` | Chrome only when a page's HTML is empty |
| One row per item | on | Off: one row per page with an `items` list |

### Output

Each result is one row in the run's dataset (download it as JSON, CSV, Excel, XML or HTML, or read it through the API). An example row (values shortened):

```json
{
  "url": "https://shop.example.com/catalogue/category/travel/",
  "pageTitle": "Travel | Example Bookshop",
  "itemIndex": 1,
  "title": "It's Only the Himalayas",
  "price": 45.17,
  "currency": "£",
  "starRating": "Two",
  "availability": "In stock",
  "link": "https://shop.example.com/catalogue/its-only-the-himalayas/",
  "scrapedAt": "2026-10-17T19:13:00Z"
}
```

From a run on 17 October 2026 on a public demo bookshop (address replaced): two category pages gave 11 and 19 books with title, price, currency, star rating, availability and link, with the same field names on both pages, for $0.015 of AI usage. A crawl of a software project's site found its latest release, the release date and the link to the release notes in 4 pages. A record named `SUMMARY` in the run's key-value store lists what each start URL returned and what was charged.

### Pricing

- **$0.008 per page** read and extracted with your prompt (or returned as Markdown)
- Free: pages that fail to load or have no readable content
- The AI model that reads the page (Google's Gemini Flash by default) is billed by Apify's AI gateway at its exact cost: about $0.006 to $0.010 a page on catalogue pages of 11 to 19 items (measured 17 October 2026); GPT and Claude cost more; Apify charges free-plan accounts 10 times the AI usage

What typical jobs cost at the list price:

| Job | Cost |
|---|---|
| One catalogue page | $0.008 + about $0.008 AI |
| 100 product pages | $0.80 + about $0.80 AI |
| Crawl a site for 10 pages | $0.08 + about $0.08 AI |
| 1,000 pages as Markdown (no prompt) | $8, no AI usage |

On Apify's Scale and Business plans the per-event prices are lower (Store discounts). Set **Maximum cost per run** and the Actor stops cleanly when it is reached: you never pay more than that.

### Tips

- List the **fields** when you need the same columns on every run; without them the first page's field names are reused on the next pages.
- Say what one item is (*every product*, *each job ad*) and the AI returns one row per item.
- In crawl mode the words of your prompt choose which links are followed: name what you are looking for.
- Leave the prompt empty to get pages as Markdown for your own model, or use the LLM Website Crawler for whole sites at $1 per 1,000 pages.

### Use it through the API

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("zaiq/ai-web-scraper").call(run_input={"startUrls": ["https://www.example.com/products"], "prompt": "Every product: name, price, rating and link"})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)
```

**JavaScript**

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('zaiq/ai-web-scraper').call({"startUrls": ["https://www.example.com/products"], "prompt": "Every product: name, price, rating and link"});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

**HTTP (one call, results back)**

```bash
curl -X POST 'https://api.apify.com/v2/acts/zaiq~ai-web-scraper/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{"startUrls": ["https://www.example.com/products"], "prompt": "Every product: name, price, rating and link"}'
```

### Integrations and schedules

- **Schedules:** run it every hour, day or week from the Apify Console (Schedules) and keep a history of results.
- **Make, Zapier, n8n, Google Sheets, Slack, Airtable:** connect it through Apify's integrations, or send each finished run to a **webhook**.
- **Datasets:** results stay in the run's dataset; export them or read them by API whenever you need them.

### Use it with AI agents (MCP)

Add Apify's MCP server (`https://mcp.apify.com`) to Claude, ChatGPT, Cursor, VS Code or any MCP client. The agent can find this Actor and run it for you: ask it for "get every product with its price and rating from this page". AI agents can also pay for runs directly through Apify's agentic payments.

### Limits

- Long pages are read in pieces of 20,000 characters, up to six (120,000 characters); items beyond that are not read.
- The AI reads what the page shows: a value that is only in an image, or behind a click, is not seen.
- Pages behind a login, CAPTCHAs and paywalls are not read.
- Like any AI, it can misread a value: check a sample before relying on a new site.

### FAQ

**Do I need an OpenAI or Google key?**

No. Apify's AI gateway runs the model and bills its exact cost to your Apify account.

**Can it read JavaScript sites?**

Yes: a page whose HTML has no content opens in Chrome (JavaScript pages: auto), or always with that setting.

**Will the columns be the same every time?**

List the fields and they are. Without fields, the AI chooses short camelCase names and keeps the first page's names for the rest of the run.

**Can it follow pagination?**

Crawl mode follows the links that match your prompt; add each page you need as a start URL for full control.

### Related Actors from Zaiq

- [llm-website-crawler](https://apify.com/zaiq/llm-website-crawler): whole sites as Markdown for LLMs
- [bulk-ai-runner](https://apify.com/zaiq/bulk-ai-runner): an AI prompt on every row of the results
- [company-research-agent](https://apify.com/zaiq/company-research-agent): company profiles from websites
- [ai-answers-api](https://apify.com/zaiq/ai-answers-api): answers from five AI assistants

### Support

Questions, a site that changed, or a field you need? Open an issue on this Actor's **Issues** tab: we answer there, usually within a day.

***

Also found as: ai web scraper, llm scraper, gpt scraper, chatgpt scraper, scrape website with ai, website to json, ai data extraction, prompt scraper, no code web scraper.

# Actor input Schema

## `startUrls` (type: `array`):

The pages to read, one per line.

## `prompt` (type: `string`):

In plain words, e.g. "every book: title, price, star rating, availability and link". Empty: each page as Markdown.

## `fields` (type: `array`):

Exact field names to return for every item, e.g. title, price, currency, url. Empty: the AI names them from the prompt.

## `mode` (type: `string`):

Crawl follows the same-site links that best match your prompt.

## `maxPagesPerSite` (type: `integer`):

How many pages to read per start URL in crawl mode ($0.008 each).

## `model` (type: `string`):

Fast is right for most pages and costs the least; the AI model's own cost is billed by Apify.

## `renderJs` (type: `string`):

Auto opens a page in a real Chrome only when its HTML has no content.

## `oneRowPerItem` (type: `boolean`):

Each extracted item as its own row (off: one row per page with an items list).

## `includeMarkdown` (type: `boolean`):

Add the cleaned page text to the first row of each page.

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
  ],
  "prompt": "Every book on the page: title, price, star rating, availability and link",
  "mode": "single",
  "maxPagesPerSite": 10,
  "model": "fast",
  "renderJs": "auto",
  "oneRowPerItem": true,
  "includeMarkdown": false
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset: the results.

## `summary` (type: `string`):

What each input returned, and notes.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
    ],
    "prompt": "Every book on the page: title, price, star rating, availability and link"
};

// Run the Actor and wait for it to finish
const run = await client.actor("zaiq/ai-web-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://books.toscrape.com/catalogue/category/books/travel_2/index.html"],
    "prompt": "Every book on the page: title, price, star rating, availability and link",
}

# Run the Actor and wait for it to finish
run = client.actor("zaiq/ai-web-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
  ],
  "prompt": "Every book on the page: title, price, star rating, availability and link"
}' |
apify call zaiq/ai-web-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zaiq/ai-web-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/DpPm5mCceB8Hihypp/builds/lwKWVPfXKPOiRWmaG/openapi.json