# AI Web Scraper: Web Page to JSON with OpenAI or Claude (`mosaicked_sponge/ai-schema-extractor`) Actor

AI web scraper that turns any web page into structured JSON. Give URLs and the fields you want (or a JSON schema) and your own OpenAI, Claude, Grok or OpenRouter key extracts validated rows, with auto-retry on bad JSON. Pay only for pages that match.

- **URL**: https://apify.com/mosaicked_sponge/ai-schema-extractor.md
- **Developed by:** [Jason](https://apify.com/mosaicked_sponge) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $15.00 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Web Scraper: Web Page to JSON with OpenAI or Claude

Turn any public web page into **clean, validated JSON** using **your own** OpenAI, Anthropic (Claude), xAI (Grok) or OpenRouter API key.
Give it URLs, list the fields you want (or paste a JSON schema), and get **one row per page** with the fields filled in.

![AI Web Scraper: Web Page to JSON with OpenAI or Claude - sample output](https://api.apify.com/v2/key-value-stores/QpfrAb6wndEDB0tDd/records/output-YDtm0vwHHsjkeLC8J.png)

> **Bring your own AI key.** This Actor does not include AI tokens. You add your own API key and pay your AI provider
> directly, at their price, for the tokens you use. A typical product or article page costs about **$0.001–0.004 in tokens**
> with a small model such as gpt-4o-mini, gpt-4.1-mini or grok-4.20-non-reasoning. No key? You still get a **free preview** (see below).

### Price

**$15 per 1,000 pages extracted** ($0.015 per page), plus Apify's tiny $0.00005 run-start fee.

You are charged **only** for pages whose AI answer was valid JSON that matched your schema (`schemaValid: true`).
**Never charged:** free preview rows, pages that failed to load, were blocked or had no text, AI errors (invalid key,
no credit, rate limit, timeout), answers that still didn't match your schema after the automatic retry, and input mistakes.

Your AI provider bills you separately for tokens. Compared with the most popular AI web scraper on the Apify Store
($20–30 per 1,000 pages with AI included), you usually pay less in total with a small model, you choose the model,
and you see the exact token usage on every row. With a large model (gpt-4o, Claude Sonnet) your token bill is higher.

### What you get

- **Field list or JSON schema**: type `price: number - price in USD`, or paste a full JSON Schema (draft 2020-12).
- **Lists**: turn on *Extract ALL matching items* to get every product, job or listing on a page as `{"items": [...]}`.
- **Validation you can trust**: every answer is checked against your schema. If the JSON is broken or a field has the
  wrong type, the model is asked once more with the exact errors. Each row reports `schemaValid` and `validationErrors`.
- **Your model, your choice**: OpenAI, Anthropic, xAI (Grok) or OpenRouter (hundreds of models). The provider is detected from
  the key (`sk-`, `sk-ant-`, `xai-`, `sk-or-`).
- **Grok strict structured output**: with an xAI key your schema is sent as a strict JSON-schema response format, so Grok's
  answer is constrained to your schema (with automatic fallback to JSON mode + validation if a schema feature isn't supported).
- **Token cap**: long pages are trimmed to *Max page tokens* (default 12,000) so a giant page can't blow up your AI bill.
  Rows show `truncated: true` when this happens.
- **Clean input for the model**: pages are converted to Markdown with menus, footers and cookie banners removed.
- **Crawl depth 1** (optional): also extract the same-site pages linked from your start URLs, up to *Max pages*.
- **Concurrency** with automatic retries on rate limits (HTTP 429, honouring Retry-After), timeouts and server errors.
- **Spending cap respected**: no AI call is made for a page that your "Maximum cost per run" could not cover,
  so you never pay tokens for pages we can't deliver.
- **Key safety**: the key is a secret input (stored encrypted by Apify), sent only in the request header to your AI
  provider, redacted from every error message, and never logged or written to the results.

### Free preview (no key)

Run it without a key and you get, for the first 2 URLs, a free row per page with **basic facts found without AI**:
title, meta description, headings, links, email addresses and JSON-LD structured data. Preview rows are labeled
`status: "preview"`, `preview: true` and are **never charged**. Add your key to get your own fields.

### Input example

```json
{
  "startUrls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
  "fields": ["title", "price: number - price in GBP", "inStock: boolean", "description - one sentence"],
  "apiKey": "sk-...",
  "model": "gpt-4o-mini"
}
```

All items on a page:

```json
{
  "startUrls": ["https://books.toscrape.com/catalogue/category/books/poetry_23/index.html"],
  "fields": ["title", "price: number", "rating - number of stars as a word"],
  "extractMany": true,
  "apiKey": "sk-ant-..."
}
```

### Output example (illustrative)

```json
{
  "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
  "status": "ok",
  "data": {"title": "A Light in the Attic", "price": 51.77, "inStock": true, "description": "A classic collection of humorous poems and drawings by Shel Silverstein."},
  "schemaValid": true,
  "validationErrors": null,
  "retried": false,
  "charged": true,
  "provider": "openai",
  "model": "gpt-4o-mini",
  "truncated": false,
  "pageTokensEstimate": 1180,
  "aiUsage": {"inputTokens": 1460, "outputTokens": 52, "calls": 1, "stopReason": "stop"}
}
```

Row `status` values: `ok` (charged), `preview`, `schema_invalid`, `ai_auth_error`, `ai_no_credit`, `ai_model_not_found`,
`ai_rate_limited`, `ai_timeout`, `ai_api_error`, `fetch_http_error`, `fetch_timeout`, `fetch_error`, `blocked_by_site`,
`blocked_url`, `no_content`, `unsupported_content`, `input_error`, `charge_limit_too_low` — all except `ok` are free.
The run's **OUTPUT** record summarises pages extracted and charged, failures by reason, AI tokens used and any pages not processed.

### Limits (honest)

- Pages are fetched over plain HTTP without a browser or proxy: sites that build content with JavaScript only, or sit behind
  Cloudflare/DataDome bot checks, return `no_content` or `blocked_by_site` (free). Server-rendered pages (most shops, blogs,
  news, docs, job boards) work well.
- PDFs, images and logins are not supported. Adult sites and private network addresses are refused.
- The AI can still misread a page. Validation guarantees the *shape* of the data, not that every value is right; spot-check
  important fields, and use *Instructions* to steer the model.

### FAQ

**Which model should I use?** Start with the default (gpt-4o-mini / claude-haiku-4-5 / grok-4.20-non-reasoning). Use a bigger model only for messy pages.

**Do you see or keep my key?** No. It's a secret input, used only for the request to your provider, never logged or stored in results.

**What if my key is wrong?** The run stops after the first rejected request with a clear `ai_auth_error` row and finishes
successfully. Nothing is charged.

**Can I use it from n8n, Make, Zapier or the API?** Yes, like any Apify Actor; pass the key in the `apiKey` input field.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to extract from, one per line (https:// is added if missing). Each page becomes one result row. Duplicates and invalid entries are skipped for free.

## `fields` (type: `array`):

One field per line: "name", "name: type" or "name: type - description". Types: string (default), number, integer, boolean, or a list such as string\[]. Example: "price: number - price in USD". Ignored when a JSON schema is given below.

## `schema` (type: `object`):

A full JSON Schema (draft 2020-12) for one page's result, e.g. {"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}},"required":\["name"]}. Shorthand {"name":"string","price":"number"} also works. Every AI answer is validated against it.

## `extractMany` (type: `boolean`):

On: the fields/schema describe ONE item and the result is {"items": \[...]} with every matching item on the page (e.g. all products on a category page, all jobs on a careers page). Still one row (and one charge) per page.

## `instructions` (type: `string`):

Extra guidance for the model, e.g. "Prices are in EUR; convert '1.299,00' to 1299.00" or "Only include products that are in stock". With no fields or schema, the model returns free-form JSON following these instructions.

## `apiKey` (type: `string`):

Your own key: sk-... (OpenAI), sk-ant-... (Anthropic), xai-... (xAI / Grok) or sk-or-... (OpenRouter). The provider is detected from the key. You pay your provider directly for tokens. Stored encrypted by Apify, sent only to your AI provider, never logged or saved in results. Leave empty for a free preview (basic page facts without AI, first 2 URLs only).

## `provider` (type: `string`):

Auto detects the provider from the key prefix. Pick one explicitly for keys without a standard prefix.

## `model` (type: `string`):

Model name at your provider. Empty = a fast, cheap default: gpt-4o-mini (OpenAI), claude-haiku-4-5 (Anthropic), grok-4.20-non-reasoning (xAI), openai/gpt-4o-mini (OpenRouter). Examples: gpt-4.1-mini, gpt-4o, claude-sonnet-4-5, grok-4.3, grok-4.7, google/gemini-2.5-flash (OpenRouter). Bigger models cost you more tokens.

## `structuredOutput` (type: `boolean`):

xAI/Grok only: send your schema as a strict JSON-schema response format so the model's answer is constrained to it. Falls back automatically to JSON mode + validation if xAI rejects a schema feature. Every answer is validated either way.

## `maxInputTokens` (type: `integer`):

Long pages are cut to this many tokens (about 4 characters each) to cap your AI cost. Rows show truncated=true when a page was cut.

## `maxOutputTokens` (type: `integer`):

Upper limit for the model's answer. Raise it for very long lists with "Extract ALL matching items".

## `crawlDepth` (type: `integer`):

0 = only the start URLs. 1 = also extract every same-site page linked from the start URLs (up to "Max pages"). Login links are skipped.

## `maxPages` (type: `integer`):

Hard cap on pages processed in one run (start URLs plus crawled pages). Also bounds your cost.

## `concurrency` (type: `integer`):

Pages processed in parallel. Lower it if your AI provider rate-limits you (HTTP 429 is retried automatically).

## `contentMode` (type: `string`):

Auto (default): main content for single pages (strips menus, footers and cookie banners to save tokens) and the full cleaned page when "Extract ALL matching items" is on, so no list item is dropped. Main content / Full page force one mode.

## `includeLinks` (type: `boolean`):

Turn on if you want the model to extract URLs (product links, apply links). Uses more tokens.

## `includeMarkdown` (type: `boolean`):

Adds the exact text sent to the model, handy for checking answers.

## `pageTimeoutSecs` (type: `integer`):

How long to wait for each web page.

## `aiTimeoutSecs` (type: `integer`):

How long to wait for the model's answer (retried once on timeout).

## `llmBaseUrl` (type: `string`):

Only for OpenAI-compatible gateways or proxies (e.g. https://my-gateway.example.com/v1). Leave empty to use the provider's official API.

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://www.python.org/about/"
  ],
  "fields": [
    "title",
    "price: number - price in GBP",
    "inStock: boolean",
    "description - one sentence"
  ],
  "extractMany": false,
  "provider": "auto",
  "structuredOutput": true,
  "maxInputTokens": 12000,
  "maxOutputTokens": 4000,
  "crawlDepth": 0,
  "maxPages": 100,
  "concurrency": 5,
  "contentMode": "auto",
  "includeLinks": false,
  "includeMarkdown": false,
  "pageTimeoutSecs": 30,
  "aiTimeoutSecs": 120
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page: data, schemaValid, validationErrors, status.

## `summary` (type: `string`):

OUTPUT record: pages extracted and charged, failures by reason, AI tokens used, pages not processed.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "https://www.python.org/about/"
    ],
    "fields": [
        "title",
        "price: number - price in GBP",
        "inStock: boolean",
        "description - one sentence"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("mosaicked_sponge/ai-schema-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "https://www.python.org/about/",
    ],
    "fields": [
        "title",
        "price: number - price in GBP",
        "inStock: boolean",
        "description - one sentence",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("mosaicked_sponge/ai-schema-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "https://www.python.org/about/"
  ],
  "fields": [
    "title",
    "price: number - price in GBP",
    "inStock: boolean",
    "description - one sentence"
  ]
}' |
apify call mosaicked_sponge/ai-schema-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mosaicked_sponge/ai-schema-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YDtm0vwHHsjkeLC8J/builds/AcI5bmC7hv7KrPuai/openapi.json