# AI Web Scraper — URL to JSON with Confidence (`crisp_gopher/ai-scraper-to-json`) Actor Extract structured data from any website into typed JSON matching your schema, with a confidence score on every field. AI-powered, RAG-ready, with built-in schema validation and grounding to catch hallucinations. - **URL**: https://apify.com/crisp\_gopher/ai-scraper-to-json.md - **Developed by:** [Emploice Mushwashans](https://apify.com/crisp_gopher) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $4.00 / 1,000 extracted results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AI Web Scraper — URL to Structured JSON with Confidence Scores **Extract structured data from any website into clean, typed JSON that matches your own schema** — with a per-field confidence score on every value. This **AI-powered web scraper** turns a URL and a JSON Schema into validated, **LLM-ready structured data**, so you know which fields to trust before you use them. Unlike other AI scrapers and **website-to-JSON** tools that hand you data and hope it's right, this actor verifies every field against the source page and tells you how confident it is: - ✅ **Your schema, enforced.** Provide a standard JSON Schema; output is validated against it and automatically repaired if the model strays. No more missing or mistyped fields. - 📊 **Per-field confidence (0–1).** Every extracted value gets a score blending the model's token probability (logprobs) with a grounding check against the page text. - 🔍 **Grounding catches hallucinations.** If a value doesn't actually appear on the page, its confidence drops — so AI-invented data is easy to filter out. - 🤖 **RAG-ready output.** Clean, typed JSON built for AI pipelines, RAG systems, training datasets, and LLM agents. - 💸 **Cheap and fast.** Lightweight HTTP fetching keeps runs inexpensive. You pay per successful result. ### What this AI scraper does Give it one or more URLs and a description of the fields you want. For each page it fetches the HTML, extracts the data your schema asks for using an LLM (OpenAI or Gemini), validates the result, and scores how trustworthy each field is. Ideal for **structured data extraction** at scale where silent errors are expensive — product data, article metadata, company info, public datasets, and more. --- ### What you get back One dataset item per URL: ```json { "url": "https://example.com/product/123", "data": { "title": "The Great Gatsby", "price": 12.99, "author": "F. Scott Fitzgerald" }, "_confidence": { "title": 0.99, "price": 1.0, "author": 0.97 }, "_grounded": true, "_meta": { "provider": "openai", "model": "gpt-4o-mini", "validation_failed": false, "repair_attempts": 0, "logprobs_available": true, "input_truncated": false, "usage": { "input_tokens": 812, "output_tokens": 34, "total_tokens": 846 } } } ```` - **`data`** — your extracted object, typed per your schema. - **`_confidence`** — score per field. `1.0` = found verbatim on the page and the model was certain; low values = weak support, treat with suspicion. - **`_grounded`** — `true` only when *every* field meets your grounding threshold. Filter your dataset on this for high-trust rows. - **`_meta`** — provider and model used, whether validation/repair fired, whether input was truncated, token usage. ### Input | Field | Required | Description | |---|---|---| | `urls` | ✅ | List of page URLs to extract from. | | `extractionSchema` | ✅ | A JSON Schema object (`type: object` with `properties`) describing what to extract. | | `provider` | | `openai` (default, supports logprobs) or `gemini` (cheapest). | | `model` | | Model for the chosen provider. Defaults: OpenAI `gpt-4o-mini`, Gemini `gemini-2.5-flash-lite`. | | `useLogprobs` | | Add the model's token-probability signal to confidence (default `true`). Auto-falls back to grounding-only on Gemini 2.5. | | `groundingThreshold` | | A field counts as grounded only at/above this score (default `0.6`). | | `openaiApiKey` | | Your OpenAI key (required when `provider` is `openai` and no built-in key is set). | | `geminiApiKey` | | Your Google AI Studio key (required when `provider` is `gemini`). | | `maxInputChars` | | Cleaned page text is truncated to this many chars before extraction, bounding token cost (default `12000`). | | `requestTimeoutSecs` | | HTTP fetch timeout per URL (default `30`). | #### Example input ```json { "urls": ["https://www.gutenberg.org/ebooks/64317"], "extractionSchema": { "type": "object", "properties": { "title": { "type": "string", "description": "Book title" }, "author": { "type": "string" }, "language": { "type": "string" } }, "required": ["title"] }, "groundingThreshold": 0.6 } ``` ### How confidence is calculated For each field, up to two signals are combined (no extra API calls): 1. **Grounding** (always on) — fuzzy string match of the extracted value against the cleaned page text (`rapidfuzz`). Catches values the model invented that aren't on the page. 2. **Logprobs** (when supported) — the model's averaged token probability for the value, from the same extraction call. Catches values the model itself was unsure about. `field_confidence = 0.6 × grounding + 0.4 × logprob` when logprobs are available, otherwise **grounding alone**. **Provider note:** OpenAI (`gpt-4o-mini`, the default) returns per-token logprobs, so you get the full combined score. Gemini's 2.5 models do **not** expose logprobs — with `provider: gemini` the actor automatically falls back to grounding-only confidence (still effective at catching hallucinations). > Grounding is a strong signal but not infallible: a value that happens to appear elsewhere on the page can score high even if it's the wrong field. Treat confidence as a strong filter, not a guarantee. ### Use cases - **AI & RAG pipelines** — build clean, typed datasets for retrieval-augmented generation and LLM agents, dropping low-confidence rows automatically. - **Training data extraction** — turn web pages into structured JSON for model training and fine-tuning. - **Price, spec & product data** — extract e-commerce and catalog data where silent errors are expensive. - **Article & content metadata** — pull titles, authors, dates, and summaries from news, blogs, and docs. - **Company & public datasets** — extract structured records from directories, government portals, and open data. - Any pipeline where "the scraper returned *something*" isn't enough — you need to know if it's **right**. ### Which websites work? This actor reads **server-rendered HTML** — it works great on the large universe of sites that ship their content in the page source: - ✅ **Works well:** news sites, blogs, documentation, GitHub, Wikipedia, government and public-data portals, company/marketing pages, forums, and many server-rendered e-commerce and listing pages. - ❌ **Not supported in v1:** sites that build content with JavaScript in the browser (e.g. Yelp, Zillow, many single-page apps) or that sit behind aggressive bot protection (Cloudflare/DataDome). These return empty or blocked responses. JavaScript rendering is planned as a future opt-in. If you're unsure, just try a single URL — the `_meta.error` field tells you immediately if a page couldn't be fetched. ### Frequently asked questions **How is this different from other AI web scrapers?** Most return raw JSON with no quality signal. This actor validates output against your JSON Schema, repairs it if needed, and attaches a confidence score to every field so you can trust — or filter — the results. **What does the confidence score mean?** A 0–1 value per field. `1.0` means the value was found on the page and the model was certain; low values mean weak support. Filter on `_grounded` to keep only high-trust rows. **Which LLM does it use?** OpenAI (`gpt-4o-mini`) by default for the richest confidence scoring, or Google Gemini for the lowest cost. You can supply your own API key. **Can it scrape Yelp / Zillow / Amazon?** Only if the page content is in the raw HTML. JavaScript-rendered and bot-protected sites are not supported in v1 (see "Which websites work?"). **Is the output ready for RAG / LLM pipelines?** Yes — output is clean, typed JSON keyed to your schema, designed to drop straight into RAG systems, vector stores, and agents. ### Limitations (v1) - **Static HTML only** — no headless browser, by design, to keep runs cheap. JavaScript-rendered sites are not yet supported. - **No proxy rotation** — heavily bot-protected sites may block requests. - Best for **page-level object extraction**, not deep multi-level crawling. # Actor input Schema ## `urls` (type: `array`): List of page URLs to extract structured data from. ## `extractionSchema` (type: `object`): A standard JSON Schema describing the object you want extracted from each page. Use "type": "object" with a "properties" map. Example: {"type":"object","properties":{"title":{"type":"string"},"price":{"type":"number"}},"required":\["title"]} ## `groundingThreshold` (type: `number`): A field is marked grounded only if its combined confidence is at or above this value. Lower = more permissive. ## `provider` (type: `string`): Which LLM provider to use. OpenAI (default) supports per-token logprobs for richer confidence scoring. Gemini is cheaper but does not expose logprobs on its 2.5 models. ## `model` (type: `string`): Model name for the chosen provider. Leave empty to use the provider default (Gemini: gemini-2.5-flash-lite, OpenAI: gpt-4o-mini). Gemini options: gemini-2.5-flash-lite, gemini-2.5-flash. OpenAI options: gpt-4o-mini. ## `useLogprobs` (type: `boolean`): Adds the model's token-probability signal to each field's confidence (on by default for quality). Supported on OpenAI; on Gemini 2.5 the actor automatically falls back to grounding-only confidence. ## `geminiApiKey` (type: `string`): Optional. Your Google AI Studio key. Required when provider is Gemini and no built-in key is configured. ## `openaiApiKey` (type: `string`): Required when provider is OpenAI. Your OpenAI API key. ## `requestTimeoutSecs` (type: `integer`): HTTP fetch timeout per URL. ## `maxInputChars` (type: `integer`): Cleaned page text is truncated to this many characters before extraction, bounding per-page token cost. Grounding still uses the full page text. The most relevant content is usually near the top. ## Actor input object example ```json { "urls": [ "https://example.com" ], "extractionSchema": { "type": "object", "properties": { "title": { "type": "string", "description": "Main heading or product title" }, "price": { "type": "number", "description": "Price as a number" } }, "required": [ "title" ] }, "groundingThreshold": 0.6, "provider": "openai", "useLogprobs": true, "requestTimeoutSecs": 30, "maxInputChars": 12000 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://example.com" ], "extractionSchema": { "type": "object", "properties": { "title": { "type": "string", "description": "Main heading or product title" }, "price": { "type": "number", "description": "Price as a number" } }, "required": [ "title" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("crisp_gopher/ai-scraper-to-json").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["https://example.com"], "extractionSchema": { "type": "object", "properties": { "title": { "type": "string", "description": "Main heading or product title", }, "price": { "type": "number", "description": "Price as a number", }, }, "required": ["title"], }, } # Run the Actor and wait for it to finish run = client.actor("crisp_gopher/ai-scraper-to-json").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://example.com" ], "extractionSchema": { "type": "object", "properties": { "title": { "type": "string", "description": "Main heading or product title" }, "price": { "type": "number", "description": "Price as a number" } }, "required": [ "title" ] } }' | apify call crisp_gopher/ai-scraper-to-json --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crisp_gopher/ai-scraper-to-json", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI Web Scraper — URL to JSON with Confidence", "description": "Extract structured data from any website into typed JSON matching your schema, with a confidence score on every field. AI-powered, RAG-ready, with built-in schema validation and grounding to catch hallucinations.", "version": "0.1", "x-build-id": "snceNaT4GRNRJsN6b" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crisp_gopher~ai-scraper-to-json/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crisp_gopher-ai-scraper-to-json", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crisp_gopher~ai-scraper-to-json/runs": { "post": { "operationId": "runs-sync-crisp_gopher-ai-scraper-to-json", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crisp_gopher~ai-scraper-to-json/run-sync": { "post": { "operationId": "run-sync-crisp_gopher-ai-scraper-to-json", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls", "extractionSchema" ], "properties": { "urls": { "title": "URLs to extract from", "type": "array", "description": "List of page URLs to extract structured data from.", "items": { "type": "string" } }, "extractionSchema": { "title": "JSON Schema of the data to extract", "type": "object", "description": "A standard JSON Schema describing the object you want extracted from each page. Use \"type\": \"object\" with a \"properties\" map. Example: {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"},\"price\":{\"type\":\"number\"}},\"required\":[\"title\"]}" }, "groundingThreshold": { "title": "Grounding threshold (0-1)", "minimum": 0, "maximum": 1, "type": "number", "description": "A field is marked grounded only if its combined confidence is at or above this value. Lower = more permissive.", "default": 0.6 }, "provider": { "title": "LLM provider", "enum": [ "gemini", "openai" ], "type": "string", "description": "Which LLM provider to use. OpenAI (default) supports per-token logprobs for richer confidence scoring. Gemini is cheaper but does not expose logprobs on its 2.5 models.", "default": "openai" }, "model": { "title": "Model", "type": "string", "description": "Model name for the chosen provider. Leave empty to use the provider default (Gemini: gemini-2.5-flash-lite, OpenAI: gpt-4o-mini). Gemini options: gemini-2.5-flash-lite, gemini-2.5-flash. OpenAI options: gpt-4o-mini." }, "useLogprobs": { "title": "Use token logprobs for confidence", "type": "boolean", "description": "Adds the model's token-probability signal to each field's confidence (on by default for quality). Supported on OpenAI; on Gemini 2.5 the actor automatically falls back to grounding-only confidence.", "default": true }, "geminiApiKey": { "title": "Gemini API key (optional override)", "type": "string", "description": "Optional. Your Google AI Studio key. Required when provider is Gemini and no built-in key is configured." }, "openaiApiKey": { "title": "OpenAI API key", "type": "string", "description": "Required when provider is OpenAI. Your OpenAI API key." }, "requestTimeoutSecs": { "title": "Request timeout (seconds)", "minimum": 5, "maximum": 120, "type": "integer", "description": "HTTP fetch timeout per URL.", "default": 30 }, "maxInputChars": { "title": "Max characters sent to the LLM", "minimum": 1000, "maximum": 100000, "type": "integer", "description": "Cleaned page text is truncated to this many characters before extraction, bounding per-page token cost. Grounding still uses the full page text. The most relevant content is usually near the top.", "default": 12000 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```