# Website Content Crawler for AI — Markdown, RAG Chunks & PDFs (`he_work/ai-website-crawler`) Actor Crawl any website into clean Markdown or text for AI, RAG and LLMs. Fast HTTP with automatic stealth-browser fallback, sitemaps and llms.txt, PDF/DOCX/PPTX/XLSX parsing, heading-aware chunks with token counts, incremental re-crawls. - **URL**: https://apify.com/he_work/ai-website-crawler.md - **Developed by:** [HE Work](https://apify.com/he_work) (community) - **Categories:** AI, Developer tools, Automation - **Security:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.55 / 1,000 pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Website Content Crawler for AI — Markdown, RAG Chunks & PDFs > ✅ **Checked daily.** An automated test runs this Actor on the Apify cloud every day and checks that it still returns real data, so if the website changes and something breaks, it's caught the same day and fixed. Crawl any website into **clean Markdown or plain text** ready for **AI, RAG pipelines, vector databases, custom GPTs and LLM fine-tuning**. ### Why this crawler - **Fast and cheap by default.** Pages are fetched over HTTP with real-browser TLS fingerprints. A stealth browser (Camoufox) starts **only for the pages that need it**, e.g. JavaScript apps or bot checks. You pay a fixed price per page, not for compute time. - **Clean content.** Main-content extraction removes menus, headers, footers, cookie banners and ads. Headings, lists, tables and links are kept as Markdown. A "whole page" mode and your own CSS selectors to remove are also available. - **Finds every page.** It reads `robots.txt` sitemaps (including sitemap indexes and `.gz` files) and `/llms.txt` as well as following links, and has include / exclude globs and scope modes. - **Documents too.** It parses linked **PDF, DOCX, PPTX, XLSX, CSV, TXT and MD** files into Markdown in the same run. - **RAG-ready.** It can split each page into heading-aware chunks with **token counts** (OpenAI `cl100k`) and a heading breadcrumb, plus a ready-to-import `chunks.jsonl`. - **Incremental re-crawls.** Give the crawl a key. Later runs mark each page `new`, `changed` or `unchanged`, skip unchanged ones (no charge), and list pages that disappeared, so you can keep a knowledge base in sync cheaply. - **One-file exports.** Each run also saves `site.md` (the whole site as one Markdown file) and `llms.txt` (a page index). - **Rich metadata.** Title, description, author, published and modified dates, language, canonical URL, site name, image, keywords, headings outline, word and token counts, content hash, optional JSON-LD and link lists. ### Input | Field | Default | Description | |---|---|---| | `startUrls` | – | Where to start (pages or direct document links) | | `crawlScope` | `path` | `path` (same folder), `host` (whole site), `domain` (+ subdomains), `single` | | `maxPages`, `maxDepth` | `100`, `20` | Limits | | `includeGlobs`, `excludeGlobs` | – | e.g. `https://example.com/docs/**`, `**/tag/**` | | `useSitemaps`, `useLlmsTxt`, `respectRobotsTxt` | `true` | Discovery | | `renderingMode` | `auto` | `auto`, `http`, `browser` | | `contentMode` | `article` | `article` (main content) or `full` | | `outputFormats` | `["markdown"]` | `markdown`, `text`, `html` | | `markdownLinks`, `markdownImages`, `removeSelectors` | `true`, `false`, – | Markdown options | | `chunkSize`, `chunkOverlap` | `0` (off), `100` | RAG chunks in tokens | | `parseDocuments` | `true` | PDF / Office / text files | | `incrementalName`, `skipUnchanged` | –, `true` | Change detection between runs | | `saveCombinedFiles` | `true` | `site.md`, `llms.txt`, `chunks.jsonl` | | `maxConcurrency`, `browserConcurrency`, `browserWaitSecs`, `requestDelayMs` | `10`, `2`, `2`, `0` | Speed / politeness | | `proxyConfiguration` | – | Optional; automatic proxy escalation is built in | ```json { "startUrls": [{ "url": "https://docs.example.com/" }], "crawlScope": "host", "maxPages": 2000, "chunkSize": 800, "incrementalName": "example-docs" } ``` ### Output (one row per page or document) `url`, `title`, `description`, `author`, `published_at`, `modified_at`, `language`, `canonical_url`, `site_name`, `image`, `keywords`, `markdown` (and/or `text`, `html`), `headings`, `word_count`, `token_count`, `content_hash`, `change_status`, `chunks` (`index`, `heading_path`, `text`, `tokens`), `content_type` (html / pdf / docx / …), `fetched_with` (http / browser), `status_code`, `depth`, `referrer`, `discovered_via` (start / link / sitemap / llms.txt), `internal_links_count`, `external_links_count`, optional `links` and `json_ld`, `crawled_at`. Key-value store: `site.md`, `llms.txt`, `chunks.jsonl`, `OUTPUT` (run report: counts, failures, removed pages). ### Example output One item from a real run (long texts shortened): ```json { "url": "https://react.dev/learn", "requested_url": "https://react.dev/learn", "status_code": 200, "depth": 0, "referrer": "", "discovered_via": "start", "fetched_with": "http", "proxy": "direct", "content_type": "html", "title": "Quick Start – React", "description": "The library for web and native user interfaces", "author": "", "published_at": "", "modified_at": "", "language": "en", "canonical_url": "https://react.dev/learn", "site_name": "", "image": "https://react.dev/images/og/learn.png", "keywords": "", "markdown": "# Quick Start – React\n\nWelcome to the React documentation! This page will give you an introduction to 80% of the React concepts that you will use on a daily …", "headings": [ { "level": 1, "text": "Quick Start – React" }, { "level": 3, "text": "You will learn" }, "…" ], "word_count": 1875, "token_count": 3339, "content_hash": "6a3fc0998df67f3a", "change_status": null, "internal_links_count": 62, "external_links_count": 23, "chunks": [ { "index": 0, "heading_path": "Quick Start – React", "text": "# Quick Start – React\n\nWelcome to the React documentation! This page will give you an introduction to 80% of the React concepts that you will use on a daily …", "tokens": 410 }, { "index": 1, "heading_path": "Quick Start – React", "text": "\\ ); } export default function MyApp() { return ( \\
\\

Welcome to my app\\

\\ \\
); } The `export default` keywords speci…", "tokens": 443 }, "…" ], "crawled_at": "2026-11-06T00:28:44+00:00" } ``` ### How it compares | | This Actor | Typical alternative | |---|---|---| | Fields per result | 29 (see *Example output*) | Varies by actor | | Freshness | Read live from the source when the run starts | Varies | | Reliability | Tested daily by an automated run on the Apify cloud (see the top of this page) | — | | Price | $3.00 per 1,000 pages | $1.56–$8.88 per 1,000 results (comparable Actors on the Apify Store, 6 Nov 2026) | ### Pricing Pay per result, no monthly fee: - **$3.00 per 1,000 pages**: One crawled page or document saved to your dataset. - **$1.50 per 1,000 browser renders** (add-on): Only for pages that needed a real browser (JavaScript apps, bot checks). - **$2.00 per 1,000 residential proxys** (add-on): Only for pages that had to be fetched through a residential IP. - $0.003 per run (start). Bronze plans pay 5% less, Silver 10% less, Gold and above 15% less. **Examples:** 1,000 pages = $3.00; 1,000 pages plus 1,000 browser renders = $4.50; 10,000 = $30.00. Fixed prices, no compute surprises: ### FAQ **How do I run it on a schedule?** Create a task with your input and add a schedule in Apify Console (e.g. daily at 6 am). Each run writes a fresh dataset. **How do I get the data?** Download the dataset as JSON, CSV, Excel, XML or HTML from the run page, or read it with the Apify API / Python and JavaScript clients. Integrations (Google Sheets, Make, Zapier, n8n, webhooks) work out of the box. **Do I need proxies or an account on the site?** No. Proxies are handled inside the Actor and included in the price; no login is needed. **What if a run returns fewer results than expected?** Check the run's key-value store `OUTPUT` record (when present) for a per-request report, then open an issue on the Issues tab with the run link and we'll look into it. **Is it legal?** The Actor only collects publicly available data. You are responsible for using it in line with the website's terms, copyright and data-protection rules (GDPR, CCPA) — don't collect personal data you have no legal basis for. ### Notes - Only crawl sites you're allowed to. robots.txt is respected by default. - Not affiliated with any of the sites you crawl. # Actor input Schema ## `startUrls` (type: `array`): Pages to start from. Links are followed within the crawl scope. Direct links to PDF / DOCX / PPTX / XLSX files work too. ## `crawlScope` (type: `string`): Which links to follow. Include / exclude globs refine it. ## `maxPages` (type: `integer`): Pages (and documents) to process per run, including ones that fail. ## `maxDepth` (type: `integer`): Clicks away from a start URL. 0 = start URLs only (plus sitemap / llms.txt pages). ## `includeGlobs` (type: `array`): Only crawl URLs matching one of these, e.g. https://example.com/docs/\*\* (\* = one path segment, \*\* = anything). ## `excludeGlobs` (type: `array`): Never crawl URLs matching these, e.g. **/tag/**, \**?page=* ## `useSitemaps` (type: `boolean`): Read robots.txt / sitemap.xml (incl. sitemap indexes and .gz) to find pages that links don't reach. ## `useLlmsTxt` (type: `boolean`): If the site publishes /llms.txt, crawl the pages it lists. ## `respectRobotsTxt` (type: `boolean`): Skip URLs disallowed for crawlers. ## `renderingMode` (type: `string`): Auto re-renders a page in a stealth browser when it is empty without JavaScript or behind a bot check. ## `contentMode` (type: `string`): How much of each page to keep. ## `outputFormats` (type: `array`): Markdown keeps headings, lists, tables and links. HTML = the raw page. ## `markdownLinks` (type: `boolean`): Off = plain link text (smaller, better for embeddings). ## `markdownImages` (type: `boolean`): Include ![alt](src) image references. ## `removeSelectors` (type: `string`): Extra elements to drop before extraction, e.g. .newsletter, #comments, .related-posts ## `chunkSize` (type: `integer`): 0 = off. Splits each page's Markdown into heading-aware chunks of up to this many LLM tokens (OpenAI cl100k tokenizer, not characters), each with its heading path. Typical: 500–1000. ## `chunkOverlap` (type: `integer`): How many LLM tokens of the previous chunk are repeated at the start of the next one. ## `parseDocuments` (type: `boolean`): Download linked PDF, DOCX, PPTX, XLSX, TXT, MD and CSV files on the site and convert them to Markdown. ## `incrementalName` (type: `string`): Any name, e.g. 'my-docs'. Runs with the same name remember page fingerprints: each page gets change_status new / changed / unchanged, and pages that disappeared are listed in the run report. ## `skipUnchanged` (type: `boolean`): With an incremental crawl name: don't output (or charge for) pages whose content didn't change since the last run. ## `includeLinkLists` (type: `boolean`): Add every internal, external and document link found on the page. ## `includeJsonLd` (type: `boolean`): Add the page's schema.org structured data. ## `saveCombinedFiles` (type: `boolean`): Also save site.md (all pages in one Markdown file), llms.txt (page index) and chunks.jsonl (vector-DB ready) in the run's key-value store. ## `maxConcurrency` (type: `integer`): Parallel HTTP requests. ## `browserConcurrency` (type: `integer`): Parallel browser pages (when a browser is needed). ## `browserWaitSecs` (type: `integer`): Extra time for JavaScript content to appear. ## `requestDelayMs` (type: `integer`): Politeness delay per host. ## `proxyConfiguration` (type: `object`): Optional. By default the crawler connects directly and moves a site to Apify datacenter, then residential proxies only if it gets blocked (included in the price). ## Actor input object example ```json { "startUrls": [ { "url": "https://quotes.toscrape.com/js/" }, { "url": "https://www.scrapingcourse.com/javascript-rendering" }, { "url": "https://react.dev/learn" } ], "crawlScope": "path", "maxPages": 20, "maxDepth": 20, "useSitemaps": true, "useLlmsTxt": true, "respectRobotsTxt": true, "renderingMode": "auto", "contentMode": "article", "outputFormats": [ "markdown" ], "markdownLinks": true, "markdownImages": false, "chunkSize": 600, "chunkOverlap": 100, "parseDocuments": true, "skipUnchanged": true, "includeLinkLists": false, "includeJsonLd": false, "saveCombinedFiles": true, "maxConcurrency": 10, "browserConcurrency": 2, "browserWaitSecs": 2, "requestDelayMs": 0 } ``` # Actor output Schema ## `pages` (type: `string`): One row per page or document. ## `siteMarkdown` (type: `string`): All pages in one Markdown file. ## `llmsTxt` (type: `string`): Index of crawled pages. ## `chunks` (type: `string`): RAG chunks (when chunk size > 0). ## `report` (type: `string`): Counts, failures, removed pages. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("he_work/ai-website-crawler").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }] } # Run the Actor and wait for it to finish run = client.actor("he_work/ai-website-crawler").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" } ] }' | apify call he_work/ai-website-crawler --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,he_work/ai-website-crawler" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/DmI9QwvVF74Byq8Dz/builds/ehORIcqEN3RmA9L6q/openapi.json