# AI Web Scraper — Extract Any Data with Claude · $0.01/page (`tidydata/ai-web-scraper`) Actor

Extract structured data from any website with AI: tell it what you need in plain words (or list the fields) and get clean JSON rows. Works on JavaScript sites, follows pagination, handles lists of items. Claude AI included, $0.01 per page.

- **URL**: https://apify.com/tidydata/ai-web-scraper.md
- **Developed by:** [tidydata](https://apify.com/tidydata) (community)
- **Categories:** AI, Automation, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.00 / 1,000 page extracted with ais

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Web Scraper: extract any data from any website with Claude — $0.01 per page

Tell it **what you need in plain words** — *"every product: name, price, rating and link"* — and get **clean JSON / CSV / Excel rows**. No selectors, no code, no breaking when the site changes its layout.

- 🧠 **Claude AI included** — no API key, no extra account.
- ⚡ **Works on JavaScript sites**: fast download first, a real browser only when the page needs it (or when a site blocks simple bots).
- 📋 **Lists or detail pages**: one row per item (products, jobs, listings, reviews…) or one row per page (a product page, a profile, an article).
- 🔗 **Follows pagination and detail links** with simple patterns like `*/page/*` or `*/product/*`.
- 💸 **$0.01 per page**, and **you only pay for pages that worked**. Pages that fail or have nothing matching are free.

### What can you extract?

Anything that is visible on the page:

- **E-commerce**: product names, prices, stock, ratings, variants, images, links.
- **Leads**: company name, email, phone, address, social links from contact or about pages.
- **Jobs**: title, company, location, salary, date, link.
- **Real estate, cars, events, directories**: every listing with its details.
- **Articles and docs**: title, author, date, summary, key facts.

### How to use it

1. Paste one or more **Start URLs**.
2. In **What to extract**, describe the data in plain words. Optionally list exact **Fields** (one per line, `price: price with currency`).
3. Choose **Rows**: *one per item* for lists, *one per page* for detail pages.
4. Optional: **Follow links matching** to open pagination or detail pages, and **Max pages** to cap the run.
5. Run. Download the results as JSON, CSV, Excel or connect them to Google Sheets, Zapier, Make or an API.

#### Example input

```json
{
  "startUrls": [{ "url": "https://books.toscrape.com/" }],
  "instruction": "Every book: title, price, star rating and whether it is in stock",
  "mode": "items",
  "followLinks": ["https://books.toscrape.com/catalogue/page-*.html"],
  "maxPages": 5
}
```

#### Example output

```json
{
  "pageUrl": "https://books.toscrape.com/",
  "pageTitle": "All products | Books to Scrape - Sandbox",
  "title": "Sapiens: A Brief History of Humankind",
  "price": "£54.23",
  "rating": "Five",
  "stock": "In stock"
}
```

### Pricing

**$0.01 per page extracted** (AI included). Failed pages and pages with nothing matching are not charged. A list page with 50 products still costs $0.01. Store discounts apply on paid Apify plans. On Apify's free plan a run reads up to 20 pages.

### Tips

- Be specific: *"price as a number without currency"* or *"date in YYYY-MM-DD"* — the AI follows your format.
- For big catalogs, start from the category page with **one row per item** and follow the pagination pattern.
- Use **Fields** when you need the same columns every run (for spreadsheets and integrations).
- The AI only uses what is on the page: missing values come back as `null`, never invented.

### FAQ

**Which AI does it use?** Claude (Anthropic), included in the price.

**Does it work on sites that need JavaScript?** Yes. In *Auto* mode it opens a real browser when the page needs it. You can also force it.

**Can it log in or solve captchas?** No. It reads public pages only. Please respect each website's terms and robots rules and the privacy laws that apply to the data you collect.

**What about very long pages?** They are cut to *Max text per page* characters (60,000 by default) and the row says so.

Related: clean and join your results with [Merge & Deduplicate Datasets](https://apify.com/tidydata/merge-dedupe-datasets), or run AI on every row with [Bulk ChatGPT & Claude](https://apify.com/tidydata/ai-dataset-enricher).

# Actor input Schema

## `startUrls` (type: `array`):

The web pages to extract data from.

## `instruction` (type: `string`):

Say in plain words what you need, e.g. 'every product: name, price, rating and link' or 'company name, email, phone and address'.

## `mode` (type: `string`):

Lists and category pages: one row per item. Product, profile or article pages: one row per page.

## `fields` (type: `array`):

Exact columns you want, one per line, optionally with a hint: 'price: price with currency'. Leave empty and the AI chooses clear column names.

## `followLinks` (type: `array`):

Also open links that match these patterns (\* = anything), e.g. 'https://books.toscrape.com/catalogue/page-*.html' for pagination or '*/product/\*' for detail pages.

## `maxPages` (type: `integer`):

Stop after this many pages (start URLs + followed links).

## `maxItemsPerPage` (type: `integer`):

For 'one row per item'.

## `renderJs` (type: `string`):

Auto detects pages that need JavaScript or block simple downloads.

## `maxCharsPerPage` (type: `integer`):

Very long pages are cut to this many characters before the AI reads them.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "instruction": "Every book on the page: title, price, star rating (1-5) and whether it is in stock.",
  "mode": "items",
  "maxPages": 10,
  "maxItemsPerPage": 200,
  "renderJs": "auto",
  "maxCharsPerPage": 60000
}
```

# Actor output Schema

## `results` (type: `string`):

One row per result.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://books.toscrape.com/"
        }
    ],
    "instruction": "Every book on the page: title, price, star rating (1-5) and whether it is in stock.",
    "mode": "items"
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidydata/ai-web-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://books.toscrape.com/" }],
    "instruction": "Every book on the page: title, price, star rating (1-5) and whether it is in stock.",
    "mode": "items",
}

# Run the Actor and wait for it to finish
run = client.actor("tidydata/ai-web-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "instruction": "Every book on the page: title, price, star rating (1-5) and whether it is in stock.",
  "mode": "items"
}' |
apify call tidydata/ai-web-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidydata/ai-web-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WdRehzqgto42IjSwA/builds/WQlhrbyx1dfw7ktMj/openapi.json