# AI Web Scraper with Playwright Browser (No-Code, MCP) (`data_rig/ai-web-scraper`) Actor Run a real Playwright browser as an AI web scraper. Extract structured data from any site using natural language—no selectors or scripts. Handles JS-heavy pages, pagination, and interactions. Built for MCP agents like OpenCode and Claude Code. - **URL**: https://apify.com/data\_rig/ai-web-scraper.md - **Developed by:** [Data Rig](https://apify.com/data_rig) (community) - **Categories:** AI, MCP servers, Automation - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 data extractions This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Playwright MCP Browser Run a real Chrome browser as an MCP-native tool so AI agents can browse, interact with, and extract data from modern websites — without writing custom scrapers. ![Playwright MCP Browser Demo](./playwright-browser-mcp-demo.gif) > Extract structured data from a live website in real time using MCP ### What this Actor is Run a real browser from an AI agent to extract data from any website — no scraper code required. Under the hood, this exposes a Playwright browser as an MCP tool. It lets agents: - load real websites (including JavaScript-heavy pages) - interact with elements (click, scroll, fill, etc.) - extract structured data, text, links, and metadata - capture screenshots All through a simple MCP interface. --- ### How it works (in 3 steps) 1. Give your agent a prompt (e.g., “extract all product listings”) 2. The browser navigates and interacts with the page 3. You get structured JSON back No selectors. No scripts. No maintenance. --- ### Why use this Actor Use this when you want **scraping workflows using natural language**. #### Key advantages - No scraper code required — works with natural language agents - Handles JavaScript-heavy sites automatically - Unified extraction (text, HTML, links, structured data) - Works with MCP-compatible agents (OpenCode, Claude Code, etc.) - Runs on Apify → scheduling, storage, APIs, proxies --- ### The shift Traditional scraping: - Write selectors - Handle JS rendering - Maintain scripts This Actor: - Describe what you want - Get structured data That’s it. --- ### Best for - AI agents that need browser access to websites - Extracting data from dynamic or JS-heavy pages - Research and content extraction workflows - QA and page inspection automation - Rapid prototyping of scraping pipelines ### Not for - Logging into websites or managing sessions - Scraping behind authentication walls - File downloads/uploads - Full-scale crawling jobs (use Apify crawlers instead) --- ### Example workflows #### 1. Competitive price monitoring - Navigate to product listing pages - Auto-detect item structure - Extract name, price, rating, URL - Paginate and repeat - Store structured dataset **Output:** ```json [{ "name": "Widget Pro", "price": 29.99, "url": "https://..." }] ```` *** #### 2. E-commerce QA automation - Load product or checkout pages - Click buttons, test inputs, navigate flows - Extract links and metadata - Capture before/after screenshots **Output:** - Pass/fail validation + screenshots per step *** #### 3. Job posting aggregator - Search job boards (LinkedIn, Indeed, etc.) - Detect job card structure automatically - Extract title, company, location, salary, URL - Combine results across multiple sources **Output:** - Unified dataset across job platforms *** ### Prompt examples (copy & run) #### Turn any page into structured data `Go to [URL], detect repeating items, and return name, price, and URL as JSON.` #### Extract all visible text from a page `Go to https://scrapethissite.com and extract all visible text, page title, and links.` #### Extract product listings `Go to https://web-scraping.dev/products and extract all products with name, price, and URL. Return structured JSON.` #### Scrape multiple pages `Go to https://web-scraping.dev/products, extract all product listings, then click next page and repeat until there are no more pages.` #### Extract specific section `Go to https://scrapingtest.com and extract only the main content area, including headings and paragraphs.` #### Aggregate job listings `Search for 'software engineer' on Indeed, extract job title, company, location, and URL, then repeat for multiple pages.` #### Test page interactions `Go to https://scrapethissite.com, click all main navigation links, take screenshots of each page, and report any broken links.` #### Extract metadata `Go to https://scrapethissite.com and extract title, meta description, OpenGraph tags, and all links.` #### Auto-detect structure and extract `Go to https://scrapingsandbox.com, detect repeating item structure, and extract structured data for each item.` *** ### Starter workflows #### Competitive price monitoring 1. Go to competitor product listing page 2. Wait for full page load 3. Detect repeating product structure 4. Extract name, price, rating, and URL 5. Click "Next page" if available 6. Repeat until pagination ends 7. Store results in dataset 8. Capture screenshot for verification Use case: - Track competitor pricing over time - Feed into analytics or alerts #### Job board aggregator 1. Go to job board (Indeed, LinkedIn, etc.) 2. Search for target role (e.g., "marketing manager") 3. Wait for results to load 4. Detect job listing structure 5. Extract title, company, location, and URL 6. Paginate through results 7. Repeat for multiple job boards 8. Combine into a unified dataset Use case: - Build lead lists from hiring signals - Identify companies actively hiring #### Website QA automation 1. Load target page 2. Capture baseline screenshot 3. Click all major navigation elements 4. Test forms and inputs 5. Extract all links 6. Identify broken links or missing metadata 7. Capture screenshots of each state 8. Output pass/fail report Use case: - Automated regression testing - SEO validation #### Content extraction pipeline 1. Go to target article or blog page 2. Extract main content (headings + paragraphs) 3. Extract metadata (title, description) 4. Extract all outbound links 5. Repeat for multiple URLs 6. Store structured content in dataset Use case: - Build datasets for AI training - Content aggregation pipelines #### Multi-site research 1. Search Google or navigate to known sources 2. Open multiple tabs 3. Extract key content from each page 4. Summarize or compare results 5. Store findings Use case: - Competitive research - Market analysis ### Recommended agent system prompt "You are a web automation agent using a browser. Navigate pages, interact when needed, and extract structured data. Prefer structured extraction when possible. Minimize unnecessary interactions." *** ### Input Configure defaults in the Input tab: ```json { "headless": true, "respectRobotsTxt": true, "userAgent": "ApifyPlaywrightMcp/1.0 (+https://apify.com)", "viewportWidth": 1280, "viewportHeight": 720, "globalTimeoutMillis": 30000, "concurrencyMode": "serialized" } ``` Only `http` and `https` URLs are supported. *** ### Output All responses are returned via MCP tool calls as structured JSON. #### Example: `page.extract` ```json { "ok": true, "result": { "text": "Example Domain...", "links": [{ "text": "Learn more", "href": "https://iana.org/domains/example" }], "metadata": { "title": "Example Domain" } }, "context": { "url": "https://example.com/" } } ``` *** ### Pricing Pay-per-event + Apify compute. | Event | When charged | Price | | --------------- | ------------------------- | ------ | | browser-session | Browser session created | $0.005 | | page-loaded | Page successfully loaded | $0.002 | | data-operation | Extraction succeeds | $0.003 | | interaction | Click, fill, scroll, etc. | $0.001 | | screenshot | Screenshot captured | $0.001 | #### Typical cost Simple workflow: - Load page → $0.002 - Extract data → $0.003 **≈ $0.005 per page** *** ### Common patterns #### Extract only what you need Use targeted extraction: ```json { "include": { "text": true, "links": true } } ``` *** #### Structured extraction ```json { "include": { "structured": { "enabled": true, "schema": { "type": "array", "itemSelector": ".product", "fields": { "name": { "selector": ".name" }, "price": { "selector": ".price" } } } } } } ``` *** #### Auto-discover structure Use: ``` page.infer_structure ``` Then pass the schema into `page.extract`. *** ### Limitations This Actor is intentionally scoped for public-web browsing. It does **not** support: - authentication or login flows - credential storage - CAPTCHA solving - file uploads/downloads - session persistence *** ### How this compares If you’ve ever written a scraper just to grab data from one page — this replaces that entire workflow. | Tool | When to use | | ---------------- | ------------------------------- | | This Actor | MCP-based browsing + extraction | | Playwright (raw) | Full custom scripting | | Apify Crawlers | Large-scale crawling jobs | | Scraping APIs | Simple static extraction | *** ### Advanced usage - Multi-tab browsing per session - Screenshots saved to key-value store - Schema-driven extraction - DNS allowlists for security - Custom Chrome binaries *** ### Roadmap This Actor is the foundation for a broader MCP-native browser ecosystem. Planned areas of expansion: - Authenticated browsing (login/session support) - Advanced interaction flows (multi-step automation) - Domain-specific extraction presets - Higher-level workflows built on top of MCP primitives The current version is intentionally scoped for reliable public-web extraction. *** ### Support - Use the Issues tab for bugs and requests - Designed for extension into a full browser MCP ecosystem *** ### Disclaimer Only scrape data you are allowed to access. Respect site terms, robots.txt, and applicable laws. # Actor input Schema ## `headless` (type: `boolean`): Run Chromium in headless mode. ## `respectRobotsTxt` (type: `boolean`): Check robots.txt before page navigation. Enabled by default. ## `userAgent` (type: `string`): Browser user agent for navigation and robots checks. ## `locale` (type: `string`): Browser locale. ## `viewportWidth` (type: `integer`): Browser viewport width in pixels. ## `viewportHeight` (type: `integer`): Browser viewport height in pixels. ## `globalTimeoutMillis` (type: `integer`): Default timeout for browser operations. ## `concurrencyMode` (type: `string`): How calls within a single MCP browser session are handled. ## `maxTextChars` (type: `integer`): Default maximum text returned from extraction. ## `maxHtmlChars` (type: `integer`): Default maximum HTML returned from extraction. ## `maxLinks` (type: `integer`): Default maximum link count returned from extraction. ## `maxScreenshotInlineBytes` (type: `integer`): Maximum screenshot size allowed when inline base64 is requested. ## `allowedHosts` (type: `array`): Optional exact Host header allowlist for DNS rebinding protection, for example \['127.0.0.1:4321']. Leave empty to rely on the Apify gateway or local trusted access. ## `allowedOrigins` (type: `array`): Optional exact Origin header allowlist for DNS rebinding protection. Leave empty to rely on the Apify gateway or local trusted access. ## Actor input object example ```json { "headless": true, "respectRobotsTxt": true, "userAgent": "ApifyPlaywrightMcp/1.0 (+https://apify.com)", "locale": "en-US", "viewportWidth": 1280, "viewportHeight": 720, "globalTimeoutMillis": 30000, "concurrencyMode": "serialized", "maxTextChars": 30000, "maxHtmlChars": 50000, "maxLinks": 200, "maxScreenshotInlineBytes": 2000000, "allowedHosts": [], "allowedOrigins": [] } ``` # Actor output Schema ## `mcp` (type: `string`): No description ## `screenshots` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("data_rig/ai-web-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("data_rig/ai-web-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call data_rig/ai-web-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=data_rig/ai-web-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI Web Scraper with Playwright Browser (No-Code, MCP)", "description": "Run a real Playwright browser as an AI web scraper. Extract structured data from any site using natural language—no selectors or scripts. Handles JS-heavy pages, pagination, and interactions. Built for MCP agents like OpenCode and Claude Code.", "version": "0.0", "x-build-id": "zMmiVTQpfMfrUItvi" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/data_rig~ai-web-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-data_rig-ai-web-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/data_rig~ai-web-scraper/runs": { "post": { "operationId": "runs-sync-data_rig-ai-web-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/data_rig~ai-web-scraper/run-sync": { "post": { "operationId": "run-sync-data_rig-ai-web-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "headless": { "title": "Headless browser", "type": "boolean", "description": "Run Chromium in headless mode.", "default": true }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Check robots.txt before page navigation. Enabled by default.", "default": true }, "userAgent": { "title": "User agent", "type": "string", "description": "Browser user agent for navigation and robots checks.", "default": "ApifyPlaywrightMcp/1.0 (+https://apify.com)" }, "locale": { "title": "Locale", "type": "string", "description": "Browser locale.", "default": "en-US" }, "viewportWidth": { "title": "Viewport width", "minimum": 320, "maximum": 3840, "type": "integer", "description": "Browser viewport width in pixels.", "default": 1280 }, "viewportHeight": { "title": "Viewport height", "minimum": 240, "maximum": 2160, "type": "integer", "description": "Browser viewport height in pixels.", "default": 720 }, "globalTimeoutMillis": { "title": "Global timeout", "minimum": 1000, "maximum": 120000, "type": "integer", "description": "Default timeout for browser operations.", "default": 30000 }, "concurrencyMode": { "title": "Concurrency mode", "enum": [ "serialized", "parallel", "reject" ], "type": "string", "description": "How calls within a single MCP browser session are handled.", "default": "serialized" }, "maxTextChars": { "title": "Max text chars", "minimum": 1000, "maximum": 500000, "type": "integer", "description": "Default maximum text returned from extraction.", "default": 30000 }, "maxHtmlChars": { "title": "Max HTML chars", "minimum": 1000, "maximum": 1000000, "type": "integer", "description": "Default maximum HTML returned from extraction.", "default": 50000 }, "maxLinks": { "title": "Max links", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Default maximum link count returned from extraction.", "default": 200 }, "maxScreenshotInlineBytes": { "title": "Max inline screenshot bytes", "minimum": 0, "maximum": 10000000, "type": "integer", "description": "Maximum screenshot size allowed when inline base64 is requested.", "default": 2000000 }, "allowedHosts": { "title": "Allowed MCP host headers", "type": "array", "description": "Optional exact Host header allowlist for DNS rebinding protection, for example ['127.0.0.1:4321']. Leave empty to rely on the Apify gateway or local trusted access.", "default": [], "items": { "type": "string" } }, "allowedOrigins": { "title": "Allowed MCP origins", "type": "array", "description": "Optional exact Origin header allowlist for DNS rebinding protection. Leave empty to rely on the Apify gateway or local trusted access.", "default": [], "items": { "type": "string" } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```