# Website Content Crawler (`worshipful_knife/website-content-crawler`) Actor Deep crawl websites and extract clean text, Markdown, or HTML for LLMs, RAG, and AI apps. Removes navigation, ads, cookie banners. Supports headless browser & HTTP. Sitemap discovery, URL scoping, file downloads. Feed ChatGPT, LangChain, LlamaIndex, Pinecone. The cheapest content crawler on Apify. - **URL**: https://apify.com/worshipful\_knife/website-content-crawler.md - **Developed by:** [kata Kuri](https://apify.com/worshipful_knife) (community) - **Categories:** AI, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $0.70 / 1,000 page scrapeds This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Website Content Crawler **Website Content Crawler** is an Apify Actor that performs deep crawls of websites and extracts clean text content from web pages. It is designed for feeding large language models (LLMs), RAG pipelines, vector databases, and AI applications with high-quality web data. ### Key Features - **Multiple crawler engines** - Adaptive mode tries headless Firefox first and automatically falls back to HTTP if the site blocks browsers. Or choose a specific engine manually. - **Clean content extraction** - Automatically removes navigation, headers, footers, cookie banners, ads, modals, and other irrelevant page elements. - **Flexible output formats** - Save content as **Markdown**, **plain text**, or **HTML**. - **Smart URL scoping** - Stays within the start URL path. Supports include/exclude glob patterns for fine-grained control. - **Sitemap discovery** - Automatically finds and parses sitemaps to discover more pages. - **Canonical URL deduplication** - Skips duplicate pages identified by the same canonical URL. - **Dynamic content support** - Wait for JavaScript rendering, scroll to trigger lazy loading, expand accordions and tabs. - **Cookie banner dismissal** - Automatically detects and dismisses cookie consent popups. - **File downloads** - Optionally download linked PDF, DOC, DOCX, XLS, XLSX, and CSV files. - **Rich metadata extraction** - Extracts title, description, author, keywords, language, and canonical URL from every page. ### Use Cases #### Feed LLMs and AI Applications Crawl documentation sites, knowledge bases, help centers, or blogs and feed the extracted content directly into your LLM, ChatGPT, or custom AI assistant. #### Retrieval Augmented Generation (RAG) Build a knowledge base from any website. Use the crawled content with vector databases like Pinecone, Qdrant, or Weaviate to power RAG-based question answering. #### Custom GPTs and AI Assistants Export crawled data as JSON and upload it as knowledge files to your custom OpenAI GPTs or AI assistants. #### Content Processing at Scale Scrape content for summarization, translation, proofreading, or style transformation using LLMs. #### LangChain and LlamaIndex Integration Use the Apify integration with LangChain or LlamaIndex to feed crawled content directly into your AI pipeline. ### How It Works The crawler operates in three stages: 1. **Crawling** - Discovers and downloads web pages starting from your URLs, following links within scope. 2. **HTML Processing** - Cleans the DOM by removing navigation, ads, cookie warnings, and other noise. 3. **Output** - Converts the cleaned HTML to your chosen format (Markdown, text, or HTML) with metadata. ### Input Configuration The only required input is **Start URLs**. All other settings have sensible defaults. | Setting | Description | Default | |---------|-------------|---------| | Start URLs | URLs to begin crawling from | (required) | | Crawler type | Engine: Adaptive, Firefox browser, or Cheerio HTTP | Adaptive | | Max pages | Maximum number of pages to crawl | 100 | | Max crawling depth | How deep to follow links from start URLs | 20 | | Output format | Markdown, plain text, or HTML | Markdown | | Exclude URLs (globs) | Glob patterns for URLs to skip | (none) | | Include URLs (globs) | Only crawl URLs matching these globs | (none) | | Remove elements (CSS) | Additional CSS selectors to remove | (none, defaults always applied) | | Extract elements (CSS) | Only keep content from these elements | (none) | | Remove cookie warnings | Auto-dismiss cookie consent banners | Yes | | Wait for dynamic content | Time to wait for JS rendering (ms) | 1000 | | Scroll height | Scroll to trigger lazy loading (px) | 0 | | Expand clickables | Click accordions/tabs to expand | No | | Save files | Download linked PDF/DOC/XLS files | No | | Use sitemaps | Discover URLs from sitemaps | Yes | ### Output Format Each crawled page produces a JSON object: ```json { "url": "https://example.com/docs/getting-started", "crawl": { "loadedUrl": "https://example.com/docs/getting-started", "loadedTime": "2024-01-15T10:30:00.000Z", "depth": 1 }, "metadata": { "canonicalUrl": "https://example.com/docs/getting-started", "title": "Getting Started | Example Docs", "description": "Learn how to get started with Example.", "author": "Example Team", "keywords": "docs, getting started", "languageCode": "en" }, "text": null, "markdown": "## Getting Started\n\nWelcome to Example...", "html": null } ```` The content field (`text`, `markdown`, or `html`) is populated based on your chosen output format. The other two fields will be `null`. ### Pricing **Only $0.001 per page** ($1.00 per 1,000 pages) via pay-per-event billing. | | This Actor | Official Apify Crawler | Firecrawl-based Actors | |---|---|---|---| | Price per page | **$0.001** | $0.005 - $0.05 | $0.004 | | 1,000 pages | **$1.00** | $5.00 - $50.00 | $4.00 | | 10,000 pages | **$10.00** | $50.00 - $500.00 | $40.00 | - **4x cheaper** than Firecrawl-based alternatives - **Up to 5x cheaper** than the official browser crawler - You only pay for pages successfully crawled and saved to the dataset Apify's free plan includes $5/month in credits, enough to crawl ~5,000 pages for free. ### Integration Examples #### Python (LangChain) ```python from langchain_community.utilities import ApifyWrapper apify = ApifyWrapper() loader = apify.call_actor( actor_id="worshipful_knife/website-content-crawler", run_input={ "startUrls": [{"url": "https://docs.example.com/"}], "maxCrawlPages": 50 }, dataset_mapping_function=lambda item: Document( page_content=item["markdown"] or item["text"] or "", metadata={"source": item["url"]} ), ) ``` #### Node.js (Apify Client) ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const run = await client.actor('worshipful_knife/website-content-crawler').call({ startUrls: [{ url: 'https://docs.example.com/' }], maxCrawlPages: 50, outputFormat: 'markdown', }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); ``` ### Troubleshooting - **Missing content?** Try switching to headless browser crawler type, which renders JavaScript. - **Too much noise in output?** Use the "Remove HTML elements" or "Extract HTML elements" CSS selectors to fine-tune. - **Crawler too slow?** Increase "Max concurrency" or switch to Cheerio crawler for static sites. - **Getting blocked?** Use the headless browser crawler type with residential proxies. ### Support If you have any questions or feedback, please open an issue on the Actor's GitHub page or contact us through Apify support. # Actor input Schema ## `startUrls` (type: `array`): One or more URLs to start crawling from. The crawler will follow links found on these pages. ## `crawlerType` (type: `string`): Select the crawler engine. ## `maxCrawlPages` (type: `integer`): Maximum number of pages to crawl. ## `maxCrawlDepth` (type: `integer`): Maximum link depth from start URLs. 0 = only start URLs. ## `maxConcurrency` (type: `integer`): Maximum number of pages crawled in parallel. ## `excludeUrlGlobs` (type: `array`): Glob patterns for URLs to skip. E.g. https://example.com/archive/\*\* ## `includeUrlGlobs` (type: `array`): If set, only URLs matching these globs will be crawled. ## `outputFormat` (type: `string`): Format of the extracted content. ## `removeElementsCssSelector` (type: `string`): CSS selector for additional elements to remove. Nav, footer, header, ads, scripts, and iframes are always removed by default. ## `extractElementsCssSelector` (type: `string`): If set, only keep content from elements matching this selector. E.g. 'article, main, .content' ## `removeCookieWarnings` (type: `boolean`): Attempt to auto-dismiss cookie consent banners. ## `waitForDynamicContent` (type: `integer`): Time in ms to wait after page load for dynamic content. Only for browser crawler. ## `maxScrollHeight` (type: `integer`): Scroll the page to this height to trigger lazy loading. 0 = no scroll. ## `expandClickables` (type: `boolean`): Click elements like accordions and tabs to expand their content. ## `saveFiles` (type: `boolean`): Download linked document files (PDF, DOC, DOCX, XLS, XLSX, CSV). ## `useSitemaps` (type: `boolean`): Discover additional URLs from sitemaps. ## `proxyConfiguration` (type: `object`): Proxy settings for the crawler. ## `requestTimeoutSecs` (type: `integer`): Maximum time for loading a single page. Lower values help avoid hanging on blocked sites. ## Actor input object example ```json { "startUrls": [ { "url": "https://docs.apify.com/" } ], "crawlerType": "playwright:adaptive", "maxCrawlPages": 100, "maxCrawlDepth": 20, "maxConcurrency": 10, "excludeUrlGlobs": [], "includeUrlGlobs": [], "outputFormat": "markdown", "removeElementsCssSelector": "", "extractElementsCssSelector": "", "removeCookieWarnings": true, "waitForDynamicContent": 1000, "maxScrollHeight": 0, "expandClickables": false, "saveFiles": false, "useSitemaps": true, "proxyConfiguration": { "useApifyProxy": true }, "requestTimeoutSecs": 30 } ``` # Actor output Schema ## `defaultDataset` (type: `string`): A dataset containing one item per crawled page. Each item includes the page URL, crawl metadata (depth, loaded time), page metadata (title, description, author, language), and the extracted content in the selected output format (text, markdown, or html). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://docs.apify.com/" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("worshipful_knife/website-content-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://docs.apify.com/" }] } # Run the Actor and wait for it to finish run = client.actor("worshipful_knife/website-content-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://docs.apify.com/" } ] }' | apify call worshipful_knife/website-content-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=worshipful_knife/website-content-crawler", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Website Content Crawler", "description": "Deep crawl websites and extract clean text, Markdown, or HTML for LLMs, RAG, and AI apps. Removes navigation, ads, cookie banners. Supports headless browser & HTTP. Sitemap discovery, URL scoping, file downloads. Feed ChatGPT, LangChain, LlamaIndex, Pinecone. The cheapest content crawler on Apify.", "version": "1.0", "x-build-id": "wPAO3szFwejVv7rqF" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/worshipful_knife~website-content-crawler/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-worshipful_knife-website-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/worshipful_knife~website-content-crawler/runs": { "post": { "operationId": "runs-sync-worshipful_knife-website-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/worshipful_knife~website-content-crawler/run-sync": { "post": { "operationId": "run-sync-worshipful_knife-website-content-crawler", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "One or more URLs to start crawling from. The crawler will follow links found on these pages.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "crawlerType": { "title": "Crawler type", "enum": [ "playwright:adaptive", "playwright:firefox", "cheerio" ], "type": "string", "description": "Select the crawler engine.", "default": "playwright:adaptive" }, "maxCrawlPages": { "title": "Max pages", "minimum": 1, "maximum": 1000000, "type": "integer", "description": "Maximum number of pages to crawl.", "default": 100 }, "maxCrawlDepth": { "title": "Max crawling depth", "minimum": 0, "maximum": 100, "type": "integer", "description": "Maximum link depth from start URLs. 0 = only start URLs.", "default": 20 }, "maxConcurrency": { "title": "Max concurrency", "minimum": 1, "maximum": 200, "type": "integer", "description": "Maximum number of pages crawled in parallel.", "default": 10 }, "excludeUrlGlobs": { "title": "Exclude URLs (globs)", "type": "array", "description": "Glob patterns for URLs to skip. E.g. https://example.com/archive/**", "default": [], "items": { "type": "string" } }, "includeUrlGlobs": { "title": "Include URLs (globs)", "type": "array", "description": "If set, only URLs matching these globs will be crawled.", "default": [], "items": { "type": "string" } }, "outputFormat": { "title": "Output format", "enum": [ "text", "markdown", "html" ], "type": "string", "description": "Format of the extracted content.", "default": "markdown" }, "removeElementsCssSelector": { "title": "Additional elements to remove (CSS selector)", "type": "string", "description": "CSS selector for additional elements to remove. Nav, footer, header, ads, scripts, and iframes are always removed by default.", "default": "" }, "extractElementsCssSelector": { "title": "Extract HTML elements (CSS selector)", "type": "string", "description": "If set, only keep content from elements matching this selector. E.g. 'article, main, .content'", "default": "" }, "removeCookieWarnings": { "title": "Remove cookie warnings", "type": "boolean", "description": "Attempt to auto-dismiss cookie consent banners.", "default": true }, "waitForDynamicContent": { "title": "Wait for dynamic content (ms)", "minimum": 0, "maximum": 30000, "type": "integer", "description": "Time in ms to wait after page load for dynamic content. Only for browser crawler.", "default": 1000 }, "maxScrollHeight": { "title": "Maximum scroll height (px)", "minimum": 0, "maximum": 100000, "type": "integer", "description": "Scroll the page to this height to trigger lazy loading. 0 = no scroll.", "default": 0 }, "expandClickables": { "title": "Expand clickable elements", "type": "boolean", "description": "Click elements like accordions and tabs to expand their content.", "default": false }, "saveFiles": { "title": "Save files", "type": "boolean", "description": "Download linked document files (PDF, DOC, DOCX, XLS, XLSX, CSV).", "default": false }, "useSitemaps": { "title": "Use sitemaps", "type": "boolean", "description": "Discover additional URLs from sitemaps.", "default": true }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy settings for the crawler.", "default": { "useApifyProxy": true } }, "requestTimeoutSecs": { "title": "Request timeout (seconds)", "minimum": 10, "maximum": 300, "type": "integer", "description": "Maximum time for loading a single page. Lower values help avoid hanging on blocked sites.", "default": 30 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```