# General Web Scraper & AI Data Extraction (`erng/general-web-scraper`) Actor Scrape any website and extract clean markdown/text content. Uses Playwright for JS rendering, Readability for article extraction, and outputs clean machine-readable data. - **URL**: https://apify.com/erng/general-web-scraper.md - **Developed by:** [Ernest Gaigulo](https://apify.com/erng) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.005 / actor start This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## General Web Scraper & AI Data Extraction [![Apify Actor](https://img.shields.io/badge/Apify-Actor-1a73e8)](https://console.apify.com/actors/fUkEIV8ta9gJ6fh9q) Scrape any website and get clean, machine-readable content — **markdown, plain text, or HTML** — ready for AI consumption. Uses a headless browser (Playwright) for JavaScript-heavy sites or raw HTTP (Cheerio) for fast, lightweight scraping. Content is automatically extracted via Mozilla Readability, stripping out navigation, ads, footers, and sidebars so you get only the main article. Perfect for **RAG pipelines, LLM training data, content analysis, knowledge base building, and research**. --- ### Features - ✅ **Universal scraping** — works on any website, JS-rendered or static - ✅ **Clean content extraction** — Mozilla Readability strips clutter (nav, ads, footers, sidebars) - ✅ **Multiple output formats** — Markdown (default), plain text, or raw HTML - ✅ **Smart crawling** — configurable depth, same-domain filtering, and max page limits - ✅ **Two crawling engines** — Playwright (headless browser, handles JS) or Cheerio (raw HTTP, 10x faster) - ✅ **Apify proxy** — automatic proxy rotation to avoid IP blocking - ✅ **Graceful error handling** — per-page failures don't crash the run - ✅ **AI-ready output** — clean markdown is ideal for RAG, LLM training, and content analysis ### Use Cases | Use case | Description | |---|---| | **RAG knowledge base** | Scrape documentation sites, blogs, or wikis and feed clean markdown into vector DBs | | **LLM training data** | Extract article content from thousands of pages for dataset creation | | **Competitive research** | Monitor competitor websites, product pages, and pricing | | **Content analysis** | Extract and analyze article content, sentiment, or topics at scale | | **Documentation archiving** | Crawl and save entire documentation sites as clean markdown | | **News monitoring** | Scrape news articles and extract the main content without ads or clutter | | **Lead generation** | Extract business data from directory sites and listings | ### Input Configuration | Field | Type | Default | Description | |---|---|---|---| | `startUrls` | `array` | Required | One or more URLs to start scraping from | | `maxPages` | `integer` | `25` | Maximum pages to scrape (0 = unlimited) | | `maxCrawlDepth` | `integer` | `2` | Maximum link depth from start URLs (0 = start URLs only) | | `sameDomain` | `boolean` | `true` | Only crawl links within the same domain | | `crawlerType` | `enum` | `playwright:chrome` | `playwright:chrome` (JS rendering) or `cheerio` (raw HTTP) | | `outputFormat` | `enum` | `markdown` | `markdown`, `text`, or `html` | | `extractMainContent` | `boolean` | `true` | Strip nav/footer/ads via Readability, or get full page content | | `proxyConfiguration` | `object` | Apify proxy | Proxy settings for IP rotation | #### Example Input ```json { "startUrls": [ { "url": "https://example.com/blog" } ], "maxPages": 50, "maxCrawlDepth": 3, "sameDomain": true, "crawlerType": "playwright:chrome", "outputFormat": "markdown", "extractMainContent": true } ```` ### Output Format Each scraped page produces a dataset item with the following structure: | Field | Type | Description | |---|---|---| | `url` | `string` | The scraped page URL | | `title` | `string` | Page title extracted from HTML | | `content` | `string` | Extracted content in requested format (markdown/text/html) | | `excerpt` | `string` | Short description or meta description | | `textLength` | `integer` | Length of the extracted text in characters | | `crawlDepth` | `integer` | Link depth from start URLs | | `crawledAt` | `string` | ISO timestamp of when the page was scraped | #### Example Output (Markdown) ```json { "url": "https://example.com/blog/article", "title": "How Web Scraping Powers AI", "content": "# How Web Scraping Powers AI\n\nWeb scraping is the foundation of modern AI training pipelines...", "excerpt": "Learn how web scraping provides the data that fuels AI models", "textLength": 3421, "crawlDepth": 1, "crawledAt": "2026-07-04T20:30:00.000Z" } ``` ### Quick Start 1. **Add start URLs** — enter the URLs you want to scrape 2. **Configure settings** — adjust max pages, depth, and output format 3. **Run the Actor** — results appear in the dataset within seconds to minutes 4. **Export results** — download as JSON, CSV, XML, or Excel ### Performance Tips - Use **Cheerio** (`crawlerType: cheerio`) for static websites — it's 10x faster and cheaper than Playwright - Set `maxCrawlDepth: 0` if you only need the start URLs themselves - Set `extractMainContent: false` if you need the full page HTML including navigation - Use `sameDomain: false` only if you need to crawl across multiple domains - For large crawls (>100 pages), start with a small test run to validate the output format ### Pricing This Actor uses **pay-per-event (PPE)** pricing: - **Per Actor start** — a small fee each time the Actor runs - **Per dataset item** — a small fee per scraped page - Platform usage costs (compute, storage, proxy) are included in the pricing See the [Monetization](#) section in Apify Console for current prices. ### Limitations - **Free tier limit**: The Apify free plan has limited platform credits. Large crawls may require upgrading your Apify plan - **JavaScript-heavy SPAs**: Some single-page applications may not render all content within the navigation timeout - **Login-required content**: This Actor does not handle authentication — use it on publicly accessible pages only - **Rate limiting**: Some websites may block aggressive crawling. Use `proxyConfiguration` and reasonable `maxPages` values ### FAQ **Q: Can I scrape sites that require login?**\ A: No. This Actor only scrapes publicly accessible pages. **Q: What's the difference between Playwright and Cheerio?**\ A: Playwright uses a real headless Chrome browser — it renders JavaScript and handles modern web apps. Cheerio makes raw HTTP requests (no JS rendering) and is 10x faster and cheaper. Use Cheerio for static HTML sites. **Q: How is the content cleaned?**\ A: Mozilla Readability (the same algorithm behind Firefox's Reader Mode) extracts the main article content, stripping navigation, ads, footers, and sidebars. **Q: Can I scrape multiple domains at once?**\ A: Yes — set `sameDomain: false` and add URLs from different domains to `startUrls`. Be mindful of rate limits. **Q: What does "crawl depth" mean?**\ A: Depth 0 = the start URLs. Depth 1 = links found on start URLs. Depth 2 = links found on depth-1 pages, etc. ### Technical Details - **Runtime**: Node.js 24 + Playwright (headless Chrome) - **SDK**: Apify SDK v3 + Crawlee v3 - **Content extraction**: Mozilla Readability via JSDOM - **Markdown conversion**: Turndown - **Docker base**: `apify/actor-node-playwright-chrome:24-1.60.0` ### Changelog #### v0.0.1 (2026-07-04) - Initial release - Playwright and Cheerio crawling engines - Readability-based content extraction - Markdown, text, and HTML output formats - Configurable crawl depth and page limits - Apify proxy integration - Graceful abort handling *** *Built with [Apify SDK](https://docs.apify.com/sdk/js) and [Crawlee](https://crawlee.dev)* # Actor input Schema ## `startUrls` (type: `array`): One or more URLs to start scraping from. The crawler will visit these pages and optionally follow links. ## `maxPages` (type: `integer`): Maximum number of pages to scrape. Set to 0 for unlimited (use with caution). ## `maxCrawlDepth` (type: `integer`): Maximum link depth from start URLs. 0 = only start URLs, 1 = start URLs + their direct links, etc. ## `sameDomain` (type: `boolean`): If enabled, only crawl links within the same domain as the start URLs. ## `crawlerType` (type: `string`): Playwright renders JavaScript (slower but handles modern sites). Cheerio is raw HTTP (faster, but no JS rendering). ## `outputFormat` (type: `string`): Format of the extracted content in the output dataset. ## `extractMainContent` (type: `boolean`): If enabled, uses Readability to extract only the main article content (strips nav, footer, ads, sidebars). Disable to get full page content. ## `proxyConfiguration` (type: `object`): Proxy settings for crawling. Use Apify proxy to avoid IP blocking. ## Actor input object example ```json { "startUrls": [ { "url": "https://apify.com" } ], "maxPages": 25, "maxCrawlDepth": 2, "sameDomain": true, "crawlerType": "playwright:chrome", "outputFormat": "markdown", "extractMainContent": true } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://apify.com" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("erng/general-web-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://apify.com" }] } # Run the Actor and wait for it to finish run = client.actor("erng/general-web-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://apify.com" } ] }' | apify call erng/general-web-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=erng/general-web-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "General Web Scraper & AI Data Extraction", "description": "Scrape any website and extract clean markdown/text content. Uses Playwright for JS rendering, Readability for article extraction, and outputs clean machine-readable data.", "version": "0.0", "x-build-id": "u5b35ReHEejwZIMmG" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/erng~general-web-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-erng-general-web-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/erng~general-web-scraper/runs": { "post": { "operationId": "runs-sync-erng-general-web-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/erng~general-web-scraper/run-sync": { "post": { "operationId": "run-sync-erng-general-web-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "One or more URLs to start scraping from. The crawler will visit these pages and optionally follow links.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxPages": { "title": "Max pages", "minimum": 0, "maximum": 10000, "type": "integer", "description": "Maximum number of pages to scrape. Set to 0 for unlimited (use with caution).", "default": 25 }, "maxCrawlDepth": { "title": "Max crawl depth", "minimum": 0, "maximum": 10, "type": "integer", "description": "Maximum link depth from start URLs. 0 = only start URLs, 1 = start URLs + their direct links, etc.", "default": 2 }, "sameDomain": { "title": "Same domain only", "type": "boolean", "description": "If enabled, only crawl links within the same domain as the start URLs.", "default": true }, "crawlerType": { "title": "Crawler type", "enum": [ "playwright:chrome", "cheerio" ], "type": "string", "description": "Playwright renders JavaScript (slower but handles modern sites). Cheerio is raw HTTP (faster, but no JS rendering).", "default": "playwright:chrome" }, "outputFormat": { "title": "Output format", "enum": [ "markdown", "text", "html" ], "type": "string", "description": "Format of the extracted content in the output dataset.", "default": "markdown" }, "extractMainContent": { "title": "Extract main content only", "type": "boolean", "description": "If enabled, uses Readability to extract only the main article content (strips nav, footer, ads, sidebars). Disable to get full page content.", "default": true }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy settings for crawling. Use Apify proxy to avoid IP blocking." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```