# AI Web Scraper (Any Website) (`neuro-scraper/ai-web-scraper`) Actor

AI Web Scraper (Any Website) SD - AI Web Scraper (Any Website) is an AI data extraction tool that extracts structured JSON from any public web page with a plain-English prompt: one row per page or per item, pagination and crawling, JS pages, Markdown and RAG chunks. Free LLM or your own key.

- **URL**: https://apify.com/neuro-scraper/ai-web-scraper.md
- **Developed by:** [Neuro Scraper](https://apify.com/neuro-scraper) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.49 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

./README.md

# Actor input Schema

## `startUrls` (type: `array`):

Pages to extract data from, e.g. https://books.toscrape.com/ or a product, article or listing page. Any public http(s) page; no login.

## `prompt` (type: `string`):

What to extract, in plain English, e.g. "Extract every product: name, price in USD, rating and product URL. Use null for fields the page does not show." Runs on every page. Empty prompt and no schema = the page is returned as Markdown (no AI call).

## `instructions` (type: `string`):

Alternative name for 'Page extraction prompt' (used by GPT scraper tasks). When both are given they are joined.

## `outputMode` (type: `string`):

'One row per item' turns every product / post / job / row the prompt describes into its own result row. 'Free-text answer' returns the model's text in 'answer'. Markdown and chunks need no AI model.

## `outputSchema` (type: `object`):

A JSON Schema the answer must follow, e.g. {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "number"}}}. For 'One row per item' give the schema of ONE item. OpenAI and OpenRouter get it as structured output; other models get it in the prompt.

## `outputFields` (type: `array`):

Simpler than a schema: the keys every answer must use, e.g. title, price, availability, url.

## `schemaDescription` (type: `string`):

Extra notes about the fields, e.g. "price without currency symbol; date as YYYY-MM-DD".

## `extractionMode` (type: `string`):

Single: each start URL plus its 'next page' links up to 'Max pages'. Crawl: follows links of the start URL's site (link selector, globs, depth). Scout / Agentic: reads the site's sitemap (or the start page's links), lets the AI pick the URLs that match the prompt, then returns their Markdown (scout) or extracts data from them (agentic).

## `maxPages` (type: `integer`):

Pages read per start URL: the start page and its next pages in single mode (the page's own 'next' link is followed), pages extracted in crawl / scout / agentic mode.

## `maxItems` (type: `integer`):

Stops after this many result rows in total (pages, items or chunks).

## `maxCrawlDepth` (type: `integer`):

Crawl mode: how many links away from the start URL the crawler goes (0 = start page only).

## `paginationSelector` (type: `string`):

CSS selector of the 'next page' link, e.g. li.next a. Empty = found automatically (rel=next, or a link labelled Next / › / »).

## `linkSelector` (type: `string`):

Crawl mode: CSS selector of the links to follow, e.g. article h3 a. Empty = every link on the page inside the start URL's site.

## `includeUrlGlobs` (type: `array`):

Crawl / scout / agentic: only follow URLs matching one of these globs, e.g. https://books.toscrape.com/catalogue/\*\*. \*\* = anything, \* = anything but /.

## `excludeUrlGlobs` (type: `array`):

Never follow URLs matching these globs, e.g. **/login**, **/cart**.

## `skipLlmGlobs` (type: `array`):

Crawl mode: pages matching these globs are only used to find links - no AI call, no result (and no charge), e.g. category pages **/category/**.

## `targetSelector` (type: `string`):

CSS selector of the part of the page sent to the AI, e.g. main, #content or .product-list. Empty = the whole page minus navigation, header, footer and cookie banners.

## `removeElementsCssSelector` (type: `string`):

Elements removed before the AI reads the page, e.g. .ads, .related-posts, #comments.

## `contentScope` (type: `string`):

'Main content only' uses an article extractor (trafilatura) and may drop product grids; keep 'Whole page' for listings.

## `pageFormatInRequest` (type: `string`):

Markdown is compact; cleaned HTML keeps short class names and attributes some values live in.

## `removeLinkUrls` (type: `boolean`):

Drops link URLs from the Markdown sent to the AI (fewer tokens, but the AI can no longer return item URLs).

## `maxContentChars` (type: `integer`):

Longer pages are cut at this length (about 4 characters per token). Raise it for long pages and large-context models.

## `includeMarkdown` (type: `boolean`):

Also saves the page content that was sent to the AI in 'markdown' (always on in Markdown and Scout output).

## `renderJavaScript` (type: `string`):

Auto loads each page over HTTP first and opens it in headless Chromium only when the HTML has (almost) no text, or when a browser-only option below is set.

## `dynamicContentWaitSecs` (type: `integer`):

Browser: extra seconds to wait after the page loads.

## `waitForSelector` (type: `string`):

Browser: wait until this CSS selector appears (max 20 s), e.g. .product-card.

## `hasInfiniteScroll` (type: `boolean`):

Browser: scroll to the bottom until no new content loads (up to 'Max scrolls') before reading the page.

## `maxScrolls` (type: `integer`):

Browser: upper limit of scrolls per page when 'Infinite scroll' is on.

## `handleCookieConsent` (type: `boolean`):

Browser: clicks a visible 'Accept' / 'Accept all' / 'I agree' button of a cookie banner. In HTTP mode cookie banners are removed from the content.

## `deviceType` (type: `string`):

Browser: desktop or a phone (Playwright's device emulation: screen, user agent, touch). Mobile pages are often lighter.

## `mobileDeviceModel` (type: `string`):

Browser, mobile: a Playwright device name, e.g. iPhone 13, Pixel 7, Galaxy S9+. Empty = the newest iPhone in Playwright's list.

## `viewportWidth` (type: `integer`):

Browser: custom window width in pixels (default 1366 desktop / the phone's own).

## `viewportHeight` (type: `integer`):

Browser: custom window height in pixels (default 900 desktop / the phone's own).

## `saveScreenshots` (type: `boolean`):

Browser: saves a full-page PNG of every page in the key-value store; the link is in 'screenshotUrl'. Turns the browser on.

## `llmProvider` (type: `string`):

Free uses Pollinations.ai's anonymous text API: no key, a small open model (GPT-OSS 20B on 2026-10-09), about one AI call every 20-30 seconds (its anonymous limit), answers up to about 1,500 tokens (about 20-30 items per page); the page text is sent to that service. With your own key the run is much faster, handles long pages, and the AI cost is billed by that provider to you.

## `llmApiKey` (type: `string`):

Your API key for the chosen provider (OpenAI sk-..., Anthropic sk-ant-..., Gemini AIza..., OpenRouter sk-or-...). Not needed for Free.

## `model` (type: `string`):

Model id, e.g. gpt-4.1-mini, claude-haiku-4-5, gemini-2.5-flash, openai/gpt-4o-mini. Empty = picked at run time from the provider's own model list (a fast, low-cost model).

## `llmBaseUrl` (type: `string`):

Only for 'Any OpenAI-compatible API': the base URL ending in /v1, e.g. https://api.groq.com/openai/v1, https://api.deepseek.com/v1, https://api.mistral.ai/v1.

## `temperature` (type: `string`):

0 = most deterministic (best for extraction). Models that do not take it get the request without it.

## `topP` (type: `string`):

Nucleus sampling, e.g. 1. Empty = the model's default.

## `frequencyPenalty` (type: `string`):

OpenAI-format models only, e.g. 0. Empty = the model's default.

## `presencePenalty` (type: `string`):

OpenAI-format models only, e.g. 0. Empty = the model's default.

## `maxOutputTokens` (type: `integer`):

Upper limit of the answer length per page. Empty = the model's default (Anthropic: 4096).

## `reasoningEffort` (type: `string`):

OpenAI-format reasoning models (gpt-5, o-series, GPT-OSS): how long the model thinks before answering. Extraction rarely needs more than low. Models that do not take it get the request without it.

## `chunkSize` (type: `integer`):

Chunks output: target chunk size (approx. tokens = characters / 4).

## `chunkOverlap` (type: `integer`):

Chunks output: tokens repeated from the end of one chunk at the start of the next.

## `respectRobotsTxt` (type: `boolean`):

Skips pages the site's robots.txt disallows for all crawlers.

## `maxConcurrency` (type: `integer`):

How many start URLs are processed at the same time (pages of one site are read one after another, politely).

## `proxyConfiguration` (type: `object`):

Residential proxies (the default) get through to most sites; pick a country when a site shows region-specific content. The AI calls do not use the proxy.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "prompt": "Extract every book on this page: title, price, availability and the link to the book page.",
  "outputMode": "items",
  "outputFields": [],
  "extractionMode": "single",
  "maxPages": 2,
  "maxItems": 100,
  "maxCrawlDepth": 1,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "skipLlmGlobs": [],
  "contentScope": "page",
  "pageFormatInRequest": "markdown",
  "removeLinkUrls": false,
  "maxContentChars": 40000,
  "includeMarkdown": false,
  "renderJavaScript": "auto",
  "dynamicContentWaitSecs": 0,
  "hasInfiniteScroll": false,
  "maxScrolls": 10,
  "handleCookieConsent": true,
  "deviceType": "desktop",
  "saveScreenshots": false,
  "llmProvider": "free",
  "temperature": "0",
  "reasoningEffort": "",
  "chunkSize": 512,
  "chunkOverlap": 64,
  "respectRobotsTxt": true,
  "maxConcurrency": 2,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://books.toscrape.com/"
        }
    ],
    "prompt": "Extract every book on this page: title, price, availability and the link to the book page.",
    "outputMode": "items",
    "outputFields": [],
    "maxPages": 2,
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "skipLlmGlobs": [],
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("neuro-scraper/ai-web-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://books.toscrape.com/" }],
    "prompt": "Extract every book on this page: title, price, availability and the link to the book page.",
    "outputMode": "items",
    "outputFields": [],
    "maxPages": 2,
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "skipLlmGlobs": [],
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("neuro-scraper/ai-web-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "prompt": "Extract every book on this page: title, price, availability and the link to the book page.",
  "outputMode": "items",
  "outputFields": [],
  "maxPages": 2,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "skipLlmGlobs": [],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call neuro-scraper/ai-web-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neuro-scraper/ai-web-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/G8CGCzTSDBeypNUep/builds/oDqpch1XgmFW0oOU8/openapi.json