# AI Web Data Extractor – Any URL or PDF to JSON, No API Key (`gazidev/ai-web-data-extractor`) Actor

Turn any web page, PDF or DOCX into structured JSON with an LLM. Describe the fields in plain English or a JSON schema; output is validated. No OpenAI/OpenRouter key: tokens are billed to your Apify account via Apify's OpenRouter Actor. $4 per 1,000 pages.

- **URL**: https://apify.com/gazidev/ai-web-data-extractor.md
- **Developed by:** [Cemal Atakli](https://apify.com/gazidev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 page extracted to jsons

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Web Data Extractor – Turn Any URL or PDF into Structured JSON (No API Key)

Give it **web pages, PDFs or DOCX files** and say **what you want in plain English** or as a **JSON schema**. It returns **clean, typed JSON for every page**, checked against your schema. You don't write selectors, need an OpenAI or OpenRouter key, or run a browser.

- **Any page layout**: product pages, job posts, company "About"/contact pages, event pages, articles, real-estate listings, research papers, price lists.
- **Listings too**: turn on *Many records per page* to get every product, job or row on a category page as an array.
- **PDF and DOCX**: text is read straight from the file (no OCR needed for text PDFs).
- **Validated output**: every answer is parsed and validated against your schema. If the JSON is invalid, the model gets one retry. Rows that still fail are **free**.
- **Pick your model**: Gemini 2.5 Flash-Lite (default, cheapest), Gemini 2.5 Flash, GPT-4.1 mini, GPT-5 mini, Claude Haiku 4.5, DeepSeek, or any OpenRouter model ID.

**$4 per 1,000 pages extracted**, plus the LLM tokens at OpenRouter prices billed to your Apify account. That comes to about $0.0005 per page with the default model (details below).

### How it works

1. Each URL is fetched over plain HTTP. robots.txt is respected and the User-Agent carries a contact address.
2. HTML is cleaned to Markdown: menus, footers and cookie banners are removed, and tables and links are kept. Text is read from PDFs and DOCX files.
3. The text and your instruction or schema go to an LLM through **Apify's own OpenRouter Actor** ([apify/openrouter](https://apify.com/apify/openrouter)).
4. The answer is parsed and lightly normalized (`"$1,299"` → `1299`, `"yes"` → `true`), then validated. If it fails validation it is retried once, then saved.

### Pricing: what you pay

| What | Price | Who bills it |
|---|---|---|
| Page extracted to valid JSON (`page-extracted`) | **$0.004** ($4 / 1,000) | This Actor |
| Actor start | $0.00005 | This Actor |
| LLM tokens | OpenRouter list price for the model you pick | **Apify, to your account**, via the apify/openrouter Actor |
| Failed pages, robots-blocked pages, JS-only pages, invalid output after retry | **$0** for this Actor (you still pay any LLM tokens already used) | – |

**About the LLM cost.** This Actor has no AI key of its own. It calls Apify's OpenRouter proxy with **your run's Apify token**, so Apify charges the exact token cost to **your Apify account**. It shows up as usage of the `apify/openrouter` Actor, separate from this Actor's per-page fee. *Max total charge per run* limits only this Actor's per-page fee, not the token cost. Use *Max pages* and *Max characters sent to the LLM* to cap token spend.

Typical token cost per page with **google/gemini-2.5-flash-lite** ($0.10 per million input tokens, $0.40 per million output tokens):

| Page | Input tokens | LLM cost (paid plan) | LLM cost (**Apify free plan: 10×**) |
|---|---|---|---|
| Product / contact page | 1–3k | ~$0.0002–0.0004 | ~$0.002–0.004 |
| Long article, 3-page PDF | ~10k | ~$0.0012 | ~$0.012 |

> **Apify free plan:** Apify's OpenRouter Actor charges free-plan users **10× the OpenRouter price**. On any paid Apify plan you pay the plain OpenRouter price. Bigger models (GPT-5 mini, Claude Haiku) cost 3–10× more per token than Flash-Lite. Each LLM answer is capped at 2,048 output tokens.

#### Compared with other AI extractors

| | This Actor | apify/ai-web-scraper |
|---|---|---|
| Price per page | $0.004 + ~$0.0005 tokens | $0.03 |
| 1,000 pages (paid plan, Flash-Lite) | **≈ $4.50** | $30 |
| PDF / DOCX URLs | Yes | – |
| JSON-schema validation + retry | Yes | – |
| Failed / invalid rows | Free | – |
| Own API key needed | No | No |

(Competitor price from its Store page, Oct 2026.)

### Input examples

**Product pages → fixed fields** (the one-click example):

```json
{
  "startUrls": [{ "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" }],
  "instruction": "Extract the book details. Price as a number in GBP.",
  "fields": ["title", "price (number)", "currency", "inStock (boolean)", "availableCount (integer): copies in stock", "description"]
}
```

**Category / search page → every item** with a JSON schema:

```json
{
  "startUrls": [{ "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html" }],
  "multipleItemsPerPage": true,
  "jsonSchema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "price": { "type": "number" },
      "inStock": { "type": "boolean" }
    },
    "required": ["title", "price"]
  }
}
```

**Crawl a site's product pages** (follow same-domain links matching a pattern):

```json
{
  "startUrls": [{ "url": "https://books.toscrape.com/catalogue/category/books/poetry_23/index.html" }],
  "followLinks": true, "maxDepth": 1, "linkFilter": "/catalogue/[^/]+_\\d+/index\\.html$", "maxPages": 50,
  "fields": ["title", "price (number)", "upc"]
}
```

**PDF papers or reports**:

```json
{
  "startUrls": [{ "url": "https://arxiv.org/pdf/1706.03762" }],
  "fields": ["title", "authors (string[])", "abstract", "year (integer)"],
  "maxPdfPages": 3
}
```

**Plain English only** (the model picks the keys): `"instruction": "Company name, what they sell, contact email, phone and address"` with `contentMode: "full"`, so footers that hold contact details are kept.

**Chain after another Actor**: set *Read URLs from an Apify dataset* to the output dataset of a crawler, Google Search or sitemap Actor. In an integration, use `{{resource.defaultDatasetId}}`.

#### Field list syntax

One field per line: `name`, `name: description` or `name (type): description`. The types are `string`, `number`, `integer`, `boolean`, `date`, `url` and `string[]` (a list). A JSON schema also accepts the shorthand `{"name": "string", "price": "number - in USD", "tags": ["string"]}`. **Any field may be `null`** when the value isn't on the page, and the model is told never to guess.

### Output

One dataset row per page:

```json
{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "status": "ok",
  "title": "A Light in the Attic | Books to Scrape - Sandbox",
  "contentType": "text/html",
  "data": {
    "title": "A Light in the Attic",
    "price": 51.77,
    "currency": "GBP",
    "inStock": true,
    "availableCount": 22,
    "description": "It's hard to imagine a world without A Light in the Attic. ..."
  },
  "model": "google/gemini-2.5-flash-lite",
  "usage": { "promptTokens": 640, "completionTokens": 120, "totalTokens": 760, "costUsd": 0.00011, "calls": 1 },
  "contentChars": 1473,
  "contentTruncated": false,
  "validationErrors": null,
  "error": null
}
```

(Illustrative values. Token counts and cost come from OpenRouter's response for your run.)

- `status`: `ok` (charged), `invalid` (output failed the schema after the retry; the raw `data` and the `validationErrors` are included; free), `failed` or `skipped` (free, with the reason in `error`), `invalidInput`.
- With *Many records per page*, `data` is an array and `itemCount` gives its length.
- `usage.costUsd` is the token cost OpenRouter reports for the page. The run summary (key-value store `OUTPUT`) adds up tokens and LLM cost for the whole run.
- The Output tab has two tables, **Extracted data** and **Pages, tokens & errors**. Export to JSON, CSV or Excel, or pull the data through the API.

### Limits and good behavior

- **robots.txt** is respected by default, including crawl-delay. Turn it off only for sites you have permission to crawl.
- **Social-media platforms are blocked**: Facebook, Instagram, LinkedIn, X/Twitter, TikTok, Reddit, YouTube, Pinterest, Threads and others. These sites need logins and their terms forbid automated collection.
- **JavaScript-only pages** (single-page apps that send almost no HTML) are detected and returned as `failed`, free. This Actor doesn't run a browser.
- Scanned PDFs without a text layer return little text (no OCR). Files are capped at 25 MB.
- Pages longer than *Max characters sent to the LLM* (default 40,000 ≈ 10k tokens) are cut, and `contentTruncated` is set to `true`.
- You are responsible for having the right to process the pages you submit. Rights in model outputs follow the model provider's terms ([OpenRouter terms](https://openrouter.ai/terms)).

### FAQ

**Do I need an OpenAI, Gemini or OpenRouter API key?** No. The LLM is called through Apify's official OpenRouter Actor with your run's Apify token, and its cost appears on your Apify bill.

**I get "OpenRouter proxy rejected this run's API token (HTTP 401/403)".** The run token wasn't accepted by `openrouter.apify.actor`. Check that your Apify account is active and has usage left (HTTP 402 means no credit). If you start the Actor through the API with a scoped or limited token, allow that token to run Actors. The run stops calling the LLM after the first such error and charges nothing.

**Which model should I use?** Start with Gemini 2.5 Flash-Lite. Switch to Gemini 2.5 Flash or GPT-5 mini for long, messy pages or where you need reasoning, such as normalizing addresses or classifying items.

**Why was a page `invalid`?** The model's JSON didn't match your schema twice, for example a required object came back as a string. Loosen the schema or try a stronger model. Invalid rows are not charged.

**Can it log in or click?** No. It reads public pages over plain HTTP. For JavaScript-heavy sites, render them with a browser-based crawler first and pass its dataset in.

**How do I cap my spend?** *Max pages* caps both fees. *Max total charge per run* caps the per-page fee. *Max characters sent to the LLM* caps tokens per page.

### Use with AI agents (Apify MCP)

Add this Actor to Claude, Cursor or any MCP client with the [Apify MCP server](https://mcp.apify.com) (`gazidev/ai-web-data-extractor`). An agent can then call it as a tool, for example: *"extract name, price and stock from these 30 product URLs as JSON"*. The agent gets validated JSON back instead of raw HTML, which saves context and tokens. From the API, run it synchronously with `run-sync-get-dataset-items` and pass `startUrls`, `fields` or `jsonSchema`, and `model`.

### Related Actors

- **Website to Markdown**: clean Markdown and RAG chunks of whole websites (the same fetch and cleanup code).
- **PDF Text Extractor / Doc to JSON**: PDF and DOCX text, tables and OCR without an LLM.

# Actor input Schema

## `startUrls` (type: `array`):

Web pages, PDFs or DOCX files to extract data from (one row per page). Social-media sites (Facebook, Instagram, LinkedIn, X, TikTok, Reddit, YouTube...) are not supported.

## `datasetId` (type: `string`):

Optional: chain this Actor after a crawler, sitemap or search Actor. In an integration use {{resource.defaultDatasetId}}.

## `datasetField` (type: `string`):

Field name in that dataset (arrays of URLs are also accepted).

## `instruction` (type: `string`):

Tell the model what to extract, e.g. 'Extract company name, contact email and phone' or 'List every job opening with title, location and salary'. Used alone, the model picks the keys; combine with Fields or a JSON schema for a fixed structure.

## `fields` (type: `array`):

One field per line: `name`, `name: description` or `name (type): description`. Types: string, number, integer, boolean, date, url, string\[] (list). Ignored when a JSON schema is given.

## `jsonSchema` (type: `object`):

A JSON Schema for one record, e.g. {"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}}}. The shorthand {"name":"string","price":"number"} also works. Output is validated against it; any value may be null when it is not on the page.

## `multipleItemsPerPage` (type: `boolean`):

For search results, category pages, job boards, tables: return every record on the page as an array in `data` (one dataset row per page, charged once).

## `model` (type: `string`):

Called through Apify's OpenRouter Actor; token cost is billed to your Apify account at OpenRouter prices (10x on the Apify free plan). See README for per-page estimates.

## `customModel` (type: `string`):

Optional: any model ID from openrouter.ai/models, e.g. `mistralai/mistral-small-3.2-24b-instruct`. Overrides the select above.

## `maxPages` (type: `integer`):

Stop after fetching this many pages. You pay only for pages extracted successfully.

## `followLinks` (type: `boolean`):

Also extract pages linked from the start URLs (same website only), up to Max pages.

## `maxDepth` (type: `integer`):

With Follow links: 1 = pages linked from the start URLs, 2 = one level further.

## `linkFilter` (type: `string`):

E.g. `/product/` or `/jobs/\d+`. Empty = all same-domain links.

## `maxContentChars` (type: `integer`):

Longer pages are cut (about 4 characters = 1 token). Lower = cheaper LLM cost.

## `maxPdfPages` (type: `integer`):

Only the first N pages of each PDF are read (text PDFs; scanned PDFs without a text layer are not OCR'd).

## `contentMode` (type: `string`):

Use Full page if the data you need is in a header, footer or sidebar (e.g. contact details).

## `includeContent` (type: `boolean`):

Adds the cleaned Markdown/text that was sent to the model (useful for auditing).

## `respectRobots` (type: `boolean`):

Skip pages disallowed by the site's robots.txt (recommended).

## `maxConcurrency` (type: `integer`):

Pages processed in parallel.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for each page or document to download before it is skipped (not charged).

## `proxyConfiguration` (type: `object`):

Optional. Not needed for most sites.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
    },
    {
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
    }
  ],
  "datasetField": "url",
  "instruction": "Extract the book details. Price as a number in GBP.",
  "fields": [
    "title",
    "price (number)",
    "currency",
    "inStock (boolean)",
    "availableCount (integer): copies in stock",
    "description"
  ],
  "multipleItemsPerPage": false,
  "model": "google/gemini-2.5-flash-lite",
  "maxPages": 2,
  "followLinks": false,
  "maxDepth": 1,
  "maxContentChars": 40000,
  "maxPdfPages": 30,
  "contentMode": "main",
  "includeContent": false,
  "respectRobots": true,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `extracted` (type: `string`):

No description

## `overview` (type: `string`):

No description

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
        },
        {
            "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
        }
    ],
    "instruction": "Extract the book details. Price as a number in GBP.",
    "fields": [
        "title",
        "price (number)",
        "currency",
        "inStock (boolean)",
        "availableCount (integer): copies in stock",
        "description"
    ],
    "maxPages": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("gazidev/ai-web-data-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html" },
        { "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html" },
    ],
    "instruction": "Extract the book details. Price as a number in GBP.",
    "fields": [
        "title",
        "price (number)",
        "currency",
        "inStock (boolean)",
        "availableCount (integer): copies in stock",
        "description",
    ],
    "maxPages": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("gazidev/ai-web-data-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
    },
    {
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
    }
  ],
  "instruction": "Extract the book details. Price as a number in GBP.",
  "fields": [
    "title",
    "price (number)",
    "currency",
    "inStock (boolean)",
    "availableCount (integer): copies in stock",
    "description"
  ],
  "maxPages": 2
}' |
apify call gazidev/ai-web-data-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gazidev/ai-web-data-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/emurIBtm11Fgf2Xbz/builds/nPhNjAWXNJfKme89f/openapi.json