# AI Web Scraper: Prompt to JSON (`kaz_kakyo/ai-web-scraper`) Actor

AI web scraper and LLM scraper: turn any web page into structured JSON with a plain-English prompt and optional JSON Schema. AI data extraction for product pages, listings, articles and directories. $4 per 1,000 pages, about 7x cheaper than browser AI scrapers. Failed or blocked pages are free.

- **URL**: https://apify.com/kaz_kakyo/ai-web-scraper.md
- **Developed by:** [Heim](https://apify.com/kaz_kakyo) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Web Scraper — Extract Structured JSON from Web Pages ($4 / 1,000 Pages)

Extract structured JSON from any list of web page URLs using a plain-English prompt and an optional JSON Schema.

For each URL in `startUrls`, the actor fetches the HTML over plain HTTP, strips navigation, scripts and boilerplate while preserving headings, lists, links and tables, sends the cleaned page text to an AI language model, validates the returned JSON object against your schema, and writes one row per URL to the default dataset.

It is designed as a low-cost alternative to browser-heavy AI scrapers ($4 per 1,000 extracted pages vs. $30 per 1,000 pages), and you only pay for pages that return validated JSON.

### Pricing (Pay Per Event)

| Event | Price | When it is charged |
|---|---|---|
| `page-extracted` | **$0.004** ($4 per 1,000 pages) | One web page fetched and turned into a validated JSON object in the dataset. |
| `apify-actor-start` | **$0.00005** per GB | Platform start event charged by Apify at run start. |

**Failed pages are free.** You are **never** charged `page-extracted` when:

- A URL is malformed, times out, or returns an HTTP error (such as 403, 404, 429 or 5xx).
- A URL points to non-HTML content (such as a PDF, image or JSON file).
- The model output fails JSON parsing or `outputSchema` validation after a repair retry.

To cap total spend on any run, set Apify's **Maximum total charge (`maxTotalChargeUsd`)** in the run options. Before each page's AI call, the actor checks that your remaining budget covers another `page-extracted` event and stops cleanly with `stopReason: "budget_reached"` in `RUN_SUMMARY` when the cap is reached.

### Input

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `startUrls` | `string[]` | Yes | 2 `books.toscrape.com` URLs | List of `http://` or `https://` web page URLs (max 1,000). |
| `prompt` | `string` | Yes | `"Extract the book title, price and number in stock"` | Plain-English instructions describing what to extract. |
| `outputSchema` | `object` | No | — | Optional JSON Schema the extracted JSON must follow. Checked keywords: `type`, `required`, `properties`, `additionalProperties`, `items`, `enum`, `const`, `minimum`/`maximum` (and exclusive), `minLength`/`maxLength` (Unicode characters), `minItems`/`maxItems`. Other keywords, including `pattern` (not evaluated, to avoid regex blow-ups), are ignored. Output that fails is retried once, then returned as a free error row. |
| `maxPages` | `integer` | No | `100` | Maximum number of URLs from `startUrls` to process (`1`–`1000`). |
| `maxConcurrency` | `integer` | No | `2` | Concurrent AI extraction requests (`1`–`3`). |

### Output

One row per processed URL is written to the default dataset:

```json
{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "finalUrl": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "status": "ok",
  "data": {
    "title": "A Light in the Attic",
    "price": "£51.77",
    "inStock": 22
  },
  "model": "glm-4.5-flash",
  "truncated": false,
  "error": null,
  "fetchedAt": "2026-10-24T12:00:00.000Z"
}
```

Dataset views:

- **`results`** — `url`, `status`, `data`, `model`
- **`errors`** — `url`, `error`

Run summary (`RUN_SUMMARY` in the default key-value store):

```json
{
  "ok": 2,
  "error": 0,
  "errorByReason": {},
  "chargedEvents": 2,
  "modelsUsed": { "glm-4.5-flash": 2 },
  "retries429": 0,
  "budgetReached": false,
  "stopReason": null,
  "complete": true
}
```

### Honest Limits & Speed

- **Plain HTTP only:** Pages are fetched with a plain HTTP `GET` request (20 s timeout, 5 MB max response body) and an honest `ApifyAIWebScraper/0.1` User-Agent. There is **no JavaScript rendering (no headless browser)**, **no logins/cookies**, and **no CAPTCHA or bot-protection bypass**. Sites that require client-side rendering or block automated HTTP requests will return an uncharged `status: "error"` row.
- **24,000-character page cap:** Cleaned page text (headings, paragraphs, lists, link text and `|`-separated table rows) is truncated to 24,000 characters before calling the model; when truncated, the row sets `"truncated": true`.
- **Speed and rate limits:** Extraction runs on free-tier GLM flash models (`glm-4.5-flash` with automatic fallback to `glm-4.7-flash` when busy) at a concurrency of 1–3. Expect roughly **5–10 seconds per page**. Under heavy upstream load, a small share of pages may still fail after exponential-backoff retries; those pages are reported as `status: "error"` and are **not charged**.
- **AI accuracy:** Extracted `data` is generated by an AI language model (`glm-4.5-flash` / `glm-4.7-flash`) and may contain mistakes or omissions. Always validate critical fields downstream.

### Data handling

- **Where your pages go:** each page's cleaned text and your prompt are sent to Z.ai's API (GLM models, operated by a Singapore company). Z.ai's API data processing terms state that it does not store the content customers send or generate, and that data is generally processed in Singapore. If you need a different provider or data location, don't use this actor for that data.
- **What the actor keeps:** only your own run's dataset and run summary in your Apify account. Pages are fetched with an honest user agent, without logins, proxies or bot-protection bypass, and private or internal network addresses are refused.
- **Model terms you inherit:** Z.ai's terms prohibit using model output to train or improve competing AI models. Don't use this actor's output for that purpose.

# Actor input Schema

## `startUrls` (type: `array`):

Public http:// or https:// web page URLs to fetch and extract JSON from. Up to 1,000 URLs per run. One dataset row is written per URL; failed or non-HTML pages become uncharged error rows.

## `prompt` (type: `string`):

Plain-English instructions describing what fields to extract from each page.

## `outputSchema` (type: `object`):

Optional JSON Schema object (top-level `properties` and `required`) that each extracted JSON object must follow.

## `maxPages` (type: `integer`):

Maximum number of URLs from `startUrls` to process in this run (default 100, max 1,000).

## `maxConcurrency` (type: `integer`):

Maximum number of concurrent LLM extraction calls (1–3, default 2).

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  ],
  "prompt": "Extract the book title, price and number in stock",
  "maxPages": 100,
  "maxConcurrency": 2
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL with status, extracted JSON data and the model used.

## `errors` (type: `string`):

URL and error details for any failed, non-HTML or unparseable pages (not charged).

## `summary` (type: `string`):

Counts of ok pages, errors by reason, charged events, models used and 429 retries.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
    ],
    "prompt": "Extract the book title, price and number in stock"
};

// Run the Actor and wait for it to finish
const run = await client.actor("kaz_kakyo/ai-web-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    ],
    "prompt": "Extract the book title, price and number in stock",
}

# Run the Actor and wait for it to finish
run = client.actor("kaz_kakyo/ai-web-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  ],
  "prompt": "Extract the book title, price and number in stock"
}' |
apify call kaz_kakyo/ai-web-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kaz_kakyo/ai-web-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/UutJddvrBaB43LR1L/builds/m8X2QxKAtTRMoxYdp/openapi.json