# AI Web Crawler (`gek0v/ai-web-crawler`) Actor Extract structured data from any website using AI. No custom selectors needed. - **URL**: https://apify.com/gek0v/ai-web-crawler.md - **Developed by:** [Angel Rojo](https://apify.com/gek0v) (community) - **Categories:** AI - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $0.00005 / actor start This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## ๐Ÿค– AI Web Scraper โ€” GPT-Powered Data Extraction > Extract structured data from **any website** using AI. No custom selectors needed โ€” just a URL and natural language instructions. Supports OpenAI, OpenRouter, LM Studio, Ollama, Groq, and any OpenAI-compatible API. [![Apify](https://img.shields.io/badge/Apify-Actor-purple)](https://console.apify.com) [![Python](https://img.shields.io/badge/Python-3.11-blue)](https://python.org) ![GPT](https://img.shields.io/badge/GPT--4o--mini-AI-green) ![License](https://img.shields.io/badge/License-MIT-green) --- ### ๐ŸŽฏ What It Does AI Web Scraper uses **GPT-4o-mini** (or GPT-4o/GPT-4.1) to intelligently extract structured data from **any** webpage. Unlike traditional scrapers that require specific CSS selectors or XPath expressions, this Actor understands **natural language instructions** and adapts to any website structure. #### โœจ Key Features - ๐Ÿง  **Natural Language Extraction** โ€” Describe what you want in plain English, GPT does the rest - ๐Ÿ”„ **Universal Compatibility** โ€” Works on any website without custom coding per site - ๐Ÿ“Š **Structured JSON Output** โ€” Returns clean, parseable data pushed to Apify Dataset - ๐Ÿ“„ **Multi-Page Support** โ€” Automatic pagination handling (up to 50 pages) - ๐Ÿš€ **Fast Processing** โ€” Pages processed in seconds with headless Playwright - ๐Ÿ”’ **Anti-Detection** โ€” Blocks images/ads, uses realistic user-agent - โšก **Multiple AI Models** โ€” gpt-4o-mini, gpt-4o, gpt-4.1 (or any OpenAI-compatible API) --- ### ๐Ÿ’ก Use Cases | Industry | What to Extract | |----------|----------------| | ๐Ÿ›’ **E-commerce** | Product names, prices, ratings, descriptions, reviews count | | ๐Ÿ  **Real Estate** | Listings, prices, locations, agent info, property details | | ๐Ÿ“ง **Lead Generation** | Company names, emails, phone numbers, social profiles | | ๐Ÿ’ผ **Job Boards** | Job titles, salaries, companies, locations, requirements | | ๐Ÿ“ฐ **Research** | Articles, papers, reviews, social media content | | ๐Ÿ” **SEO** | Meta tags, headings, content structure, internal links | --- ### ๐Ÿ“ฅ Input Schema | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `url` | `string` | โœ… | โ€” | Target URL to scrape | | `prompt` | `string` | โœ… | โ€” | What data to extract (natural language) | | `apiKey` | `string` | โŒ | env `OPENAI_API_KEY` | OpenAI API key (`sk-...`) | | `model` | `string` | โŒ | `gpt-4o-mini` | AI model: `gpt-4o-mini`, `gpt-4o`, `gpt-4.1` | | `maxPages` | `integer` | โŒ | `1` | Max pages to process (1โ€“50) | | `waitForSelector` | `string` | โŒ | โ€” | CSS selector to wait for before extracting | #### Example Input ```json { "url": "https://www.example.com/products", "prompt": "Extract all product names, prices, ratings, and review counts", "model": "gpt-4o-mini", "maxPages": 3 } ```` *** ### ๐Ÿ“ค Output Each extracted item is pushed to the Apify **Dataset** as a separate record with these standard fields: | Field | Type | Description | |-------|------|-------------| | `title` | `string` | Title or name of the extracted item | | `description` | `string` | Description or summary | | `price` | `string` | Price value if available | | `url` | `string` | Source URL of the item | | `image_url` | `string` | Image URL if available | | `rating` | `number` | Rating score (0โ€“5 scale) | | `reviews_count` | `integer` | Number of reviews | | `availability` | `string` | Availability status | | `category` | `string` | Category or type | | `source_page` | `string` | Page where item was found | | `extracted_at` | `datetime` | ISO timestamp of extraction | > โš ๏ธ **Note:** Field names are dynamic โ€” GPT determines them based on your prompt. The schema above covers common extraction patterns for products/listings. #### Example Output ```json [ { "title": "Wireless Headphones Pro", "price": "$79.99", "rating": 4.5, "reviews_count": 1234, "url": "https://example.com/products/wireless-headphones-pro" }, { "title": "Bluetooth Speaker", "price": "$49.99", "rating": 4.2, "reviews_count": 856, "url": "https://example.com/products/bluetooth-speaker" } ] ``` *** ### ๐Ÿงช How to Use #### Option 1: Run via Apify Console 1. Go to [Apify Console](https://console.apify.com) 2. Find **"AI Web Scraper"** in the Store 3. Click **"Try for free"** or **"Run Actor"** 4. Enter your URL and extraction prompt 5. Click **"Run"** โ€” results appear in the Dataset #### Option 2: Run via API ```bash curl -X POST "https://api.apify.com/v2/acts/gek0v~ai-web-scraper/runs" \ -H "Authorization: Bearer YOUR_APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/products", "prompt": "Extract product names and prices", "model": "gpt-4o-mini" }' ``` #### Option 3: Python SDK ```python from apify_client import ApifyClient client = ApifyClient("your-apify-token") run = client.actor("gek0v/ai-web-scraper").call(run_input={ "url": "https://example.com", "prompt": "Extract all article titles and authors", "model": "gpt-4o-mini" }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` *** ### ๐Ÿ’ฐ Pricing | Component | Cost | |-----------|------| | **Actor Compute (Actor Start)** | ~$0.000002/run (based on memory allocation) | | **Dataset Storage** | ~$0.002 per stored item | | **Platform Fee** | 20% of compute + storage costs | | **OpenAI GPT API** | Passed directly to user at model pricing | > ๐Ÿ’ก **Typical cost per run:** Most extractions cost **< $0.01** (with gpt-4o-mini) plus ~$0.002 per extracted item stored. *** ### ๐Ÿ”ง Local Development ```bash ## Clone git clone https://github.com/gek0v/ai-web-scraper.git cd ai-web-scraper ## Install dependencies pip install -r requirements.txt ## Run locally python src/main.py --input '{"url": "https://example.com", "prompt": "Extract all headings"}' ``` *** ### ๐Ÿ“ Tips for Best Results 1. **Be specific in your prompt** โ€” *"Extract product name, price in USD, and star rating"* works better than *"extract product info"* 2. **Test with gpt-4o-mini first** โ€” It's 10x cheaper and often good enough. Upgrade to gpt-4o for complex pages 3. **Use `waitForSelector`** โ€” For dynamic SPAs (React, Vue, Angular), wait for the content container 4. **Limit `maxPages`** โ€” Start with 1 page to test, then scale up 5. **Provide your API key** โ€” Set `OPENAI_API_KEY` env var or pass via input *** ### โš ๏ธ Limitations - Very large pages (>100K chars) are truncated to fit GPT's context window - JavaScript-heavy SPAs may need `waitForSelector` for rendering - Some anti-bot protections (Cloudflare, etc.) may block access - GPT costs are passed through to the user (OpenAI/compatible API pricing applies) - Requires an OpenAI-compatible API key (not included) *** ### ๐Ÿ“„ License MIT License โ€” free to use and modify. *** ### ๐Ÿท๏ธ Tags `web-scraping` `artificial-intelligence` `data-extraction` `playwright` `gpt` `automation` `developer-tools` # Actor input Schema ## `url` (type: `string`): The URL of the webpage to scrape ## `prompt` (type: `string`): Describe in natural language what data to extract (e.g. 'Extract all product names, prices, and ratings') ## `apiKey` (type: `string`): API key for your AI provider. OpenAI (sk-...), OpenRouter (sk-or-...), or any compatible key. ## `baseUrl` (type: `string`): Base URL for any OpenAI-compatible API. Examples: OpenAI=https://api.openai.com/v1, OpenRouter=https://openrouter.ai/api/v1, LM Studio=http://localhost:1234/v1, Ollama=http://localhost:11434/v1, Groq=https://api.groq.com/openai/v1, Together=https://api.together.xyz/v1, Fireworks=https://api.fireworks.ai/inference/v1, or any custom endpoint. ## `model` (type: `string`): Model identifier. For OpenRouter use format 'provider/model' (e.g. 'openai/gpt-4o-mini', 'anthropic/claude-sonnet-4'). For local models use the model name from your server. ## `maxPages` (type: `integer`): Maximum number of pages to process (1-50) ## `waitForSelector` (type: `string`): CSS selector to wait for before extracting (useful for dynamic content) ## Actor input object example ```json { "url": "https://example.com/products", "prompt": "Extract all product names, prices, and ratings from the page", "baseUrl": "https://openrouter.ai/api/v1", "model": "openai/gpt-4o-mini", "maxPages": 1, "waitForSelector": ".product-list" } ``` # Actor output Schema ## `dataset` (type: `string`): Array of data items extracted by GPT based on your natural language prompt. Field names are determined dynamically by GPT. ## `keyValueStore` (type: `string`): Summary metadata including total items extracted, model used, and target URL. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("gek0v/ai-web-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("gek0v/ai-web-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call gek0v/ai-web-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=gek0v/ai-web-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI Web Crawler", "description": "Extract structured data from any website using AI. No custom selectors needed.", "version": "1.0", "x-build-id": "nPHwZOThk74h06sTW" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/gek0v~ai-web-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-gek0v-ai-web-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/gek0v~ai-web-crawler/runs": { "post": { "operationId": "runs-sync-gek0v-ai-web-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/gek0v~ai-web-crawler/run-sync": { "post": { "operationId": "run-sync-gek0v-ai-web-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "url", "prompt" ], "properties": { "url": { "title": "Target URL", "type": "string", "description": "The URL of the webpage to scrape" }, "prompt": { "title": "Extraction Instructions", "type": "string", "description": "Describe in natural language what data to extract (e.g. 'Extract all product names, prices, and ratings')" }, "apiKey": { "title": "API Key", "type": "string", "description": "API key for your AI provider. OpenAI (sk-...), OpenRouter (sk-or-...), or any compatible key." }, "baseUrl": { "title": "API Base URL", "type": "string", "description": "Base URL for any OpenAI-compatible API. Examples: OpenAI=https://api.openai.com/v1, OpenRouter=https://openrouter.ai/api/v1, LM Studio=http://localhost:1234/v1, Ollama=http://localhost:11434/v1, Groq=https://api.groq.com/openai/v1, Together=https://api.together.xyz/v1, Fireworks=https://api.fireworks.ai/inference/v1, or any custom endpoint." }, "model": { "title": "AI Model", "type": "string", "description": "Model identifier. For OpenRouter use format 'provider/model' (e.g. 'openai/gpt-4o-mini', 'anthropic/claude-sonnet-4'). For local models use the model name from your server.", "default": "gpt-4o-mini" }, "maxPages": { "title": "Max Pages", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of pages to process (1-50)", "default": 1 }, "waitForSelector": { "title": "Wait for Selector", "type": "string", "description": "CSS selector to wait for before extracting (useful for dynamic content)" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```