Docs to Markdown: Documentation Site Crawler for LLM & RAG

Turn any documentation site into clean, LLM-ready Markdown for RAG, AI agents and knowledge bases. Give it a docs URL. It finds every page through links, sitemap.xml and llms.txt, and returns one record per page with the article body as GitHub-flavoured Markdown (sidebar, nav and buttons removed), plus a .md file per page with YAML front matter. Set a State key and run it on a schedule: each run returns only the pages that are new or changed, and you pay only for those.

Input	One or more docs URLs. Scope = same host + same path prefix (https://docs.example.com/guide/ never wanders into /blog/)
Output	Title, URL, breadcrumbs, headings outline, Markdown body, plain text, word count, content hash; .md files + INDEX.json in the key-value store
Price	$0.002 per page, $0.005 per run. A 300-page docs site ≈ $0.61 once; a daily watch run with 5 changed pages ≈ $0.015
Speed	About 20 pages in 6 seconds on 256 MB. Plain HTTP, no browser

{
    "url": "https://docs.apify.com/actors/publishing/publish",
    "title": "Publish your Actor",
    "breadcrumbs": ["Actors", "Publish"],
    "markdown": "By publishing your Actor, you make it available to the public on [Apify Store](https://apify.com/store)...\n\n## Before you start\n\n...",
    "wordCount": 377,
    "changeType": "new"
}

Jump to the task you have:

1. Download a whole documentation site as Markdown files
2. Keep a RAG index or vector database in sync with the docs
3. Give an AI agent or coding assistant a product's docs as context
4. Export only one section or one version of the docs
5. Audit your own docs site
6. What this Actor cannot do

1. Download a whole documentation site as Markdown files

Generic website crawlers return the whole page: sidebar, top nav, "Edit this page", "Copy for LLM" buttons, heading anchor icons, pagination and cookie banners. This Actor finds the article body first and drops the rest. It knows the layouts of Docusaurus, MkDocs Material, Sphinx / ReadTheDocs, VitePress, Nextra, Mintlify and GitBook, and falls back to article / main on other sites. Code blocks keep their language (ts, bash, ...), tables stay tables, admonitions become blockquotes, and links and images become absolute URLs.

{ "startUrls": [{ "url": "https://docs.apify.com/actors" }], "maxPages": 300, "saveMarkdownFiles": true }

Every page is also written to the run's key-value store as page-<host>--<path>.md with front matter, plus INDEX.json (all pages with title, file key, word count and hash). Download the store, or the dataset as JSON, CSV or Excel, and drop the files into Obsidian, a Claude Code / LLM wiki or a Notion import. Aliases (/ vs /index.html, canonical URLs) and byte-identical pages are stored and billed once.

2. Keep a RAG index or vector database in sync with the docs

A docs site changes a few pages a week. Re-exporting 300 pages every day to find them is slow and you pay for every page. Watch mode compares for you:

1. Set State key (stateKey) to one name per site, e.g. acme-docs-watch, and set maxPages above the size of the site.
2. Run once. Every page is exported with "changeType": "new" and its hash is saved in a named key-value store with that name.
3. Schedule the same input daily or weekly in Apify Schedules. Each run re-crawls the site and:
   • exports and bills a page only when it is new or its Markdown changed ("changeType": "changed", plus previousContentHash);
   • skips unchanged pages for free;
   • writes CHANGES.json to the run's key-value store with the URLs that are new, changed and removed.

{
    "stateStore": "apify-docs-watch",
    "crawlComplete": true,
    "new": [],
    "changed": ["https://docs.apify.com/actors/running"],
    "removed": ["https://docs.apify.com/actors/running/old-page"],
    "unchangedCount": 11,
    "crawledAt": "2026-09-25T04:03:38.439Z"
}

Upsert the dataset items into your index by url and delete the removed URLs. Sections are already split by headings, so you can chunk by heading instead of by character count. A page counts as removed when it was exported before, lies inside the current scope and was not reached by this run. Removed pages are reported only after a complete crawl: if the run stops early because of maxPages or the run budget (crawlComplete: false), nothing is reported as removed, so a partial crawl never deletes pages from your index.

3. Give an AI agent or coding assistant a product's docs as context (MCP)

Agents connected to the Apify MCP server can call this Actor by name (vellumlabs/docs-to-markdown-corpus) with a docs URL and read the dataset. Each record is small and self-describing (title, breadcrumbs, headings, markdown), so an agent can read the outline first and fetch only the pages it needs. Keep maxPages low (20 to 50) for interactive use; the cost is capped by maxPages × $0.002.

4. Export only one section or one version of the docs

The start URL sets the scope: https://docs.example.com/api/ exports only pages under /api/. To be explicit, use globs:

{
    "startUrls": [{ "url": "https://docs.example.com/latest/" }],
    "includeUrlGlobs": ["https://docs.example.com/latest/**"],
    "excludeUrlGlobs": ["https://docs.example.com/latest/changelog/**", "https://docs.example.com/v1/**"]
}

If the wrong block is picked up as the article, pass contentSelector (e.g. .theme-doc-markdown, #main-content) and add stray elements to removeSelectors. Start with maxPages: 50 to check the output, then raise it.

5. Audit your own docs site

Each record has wordCount, headings, breadcrumbs, section, depth and internalLinks. Sort by wordCount to find stub pages, by depth to find pages buried too deep, and compare contentHash to find duplicated content. SUMMARY.json counts stored, duplicate, failed and skipped pages for the whole run.

What this Actor cannot do

Read this before your first run. A failing page never fails the whole run: it is retried, counted in pagesFailed and not billed.

Limit	What happens	What to do
JavaScript-only docs (client-side apps with no server-rendered HTML)	The HTML has no article text. The page is skipped as empty and not billed	Most docs frameworks server-render. For pure SPAs use a browser-based crawler
Login-protected or private docs	Only what a logged-out visitor sees is fetched. There is no cookie, header or credential input	None
Sites that block bots (Cloudflare challenge, 403)	The page fails after 2 retries, is listed in pagesFailed and not billed	Turn on Apify Proxy in proxyConfiguration
robots.txt disallows a path	Those URLs are never fetched (on by default), even if linked	Set respectRobotsTxt: false only if you have the right to fetch them, e.g. your own site
Pages that are neither linked nor listed	Pages are found through in-scope links, sitemap.xml (robots.txt Sitemap: lines, /sitemap.xml, /sitemap_index.xml, /docs/sitemap.xml-style paths, indexes, .xml.gz) and llms.txt. A page in none of these is not found. At most 25 sitemap files are read per run	Add those pages to startUrls
PDFs, images, downloads, OpenAPI JSON/YAML files	Never followed (binary, feed, search and data files are excluded by default)	Use a dedicated PDF or OpenAPI tool
Content inside tabs, accordions or iframes	Content that is loaded by JavaScript after a click, or shown inside an iframe, is not in the HTML and is missing	—
Pages that print a timestamp or random token in the body	In watch mode they show up as changed (and are billed) on every run	Strip that element with removeSelectors
A page still exists but is no longer linked or listed in the sitemap	Watch mode reports it as removed	Add it to startUrls if you still need it
Rate limits	Default 240 requests per minute, 8 in parallel, 60 s per page; 429 / 5xx are retried twice, then counted in pagesFailed	Lower maxConcurrency and maxRequestsPerMinute for small sites
Very large sites	maxPages caps stored (billed) pages, up to 100,000. A run cut by maxPages sets crawlComplete: false and reports no removed pages	Raise maxPages above the site size for watch mode
Very large state	The State key keeps the 50,000 most recently seen pages; older ones are forgotten and would be exported again as new	Use one State key per site or section
Budget cap	With Max total charge per run set, the run stops cleanly before going over it. Pages already billed are saved to the state (also when you abort the run), so the next run does not bill them again	Raise the cap or lower maxPages

Input

Field	Type	Default	Notes
startUrls	array	required	Docs pages to start from. Scope = same host + same path prefix.
maxPages	integer	200	Hard cap on stored pages (1 to 100,000). Each stored page is one page-scraped event, so this caps your cost.
maxCrawlDepth	integer	20	Link hops from the start URLs. 0 = start URLs only.
discoverFromSitemaps	boolean	true	Also queue in-scope pages listed in the site's sitemaps (found via robots.txt and the usual paths).
discoverFromLlmsTxt	boolean	true	Also queue in-scope links from llms.txt; page.md links are mapped to the HTML page.
stateKey	string	""	Watch mode. Name of a key-value store that remembers page hashes, e.g. acme-docs-watch. Empty = one-off export.
includeUrlGlobs	string[]	[]	Override the auto scope, e.g. https://docs.example.com/**.
excludeUrlGlobs	string[]	[]	Never follow these. Binary files, feeds and search pages are always excluded.
contentSelector	string	""	CSS selector of the main content. Empty = auto-detect.
removeSelectors	string[]	[]	Extra selectors to strip (on top of the built-in noise list).
saveMarkdownFiles	boolean	true	Also write page-<slug>.md files and an INDEX.json manifest to the key-value store.
includeHtml	boolean	false	Add cleaned content HTML to each item.
respectRobotsTxt	boolean	true	Skip URLs disallowed by robots.txt.
maxConcurrency	integer	8	Parallel requests.
maxRequestsPerMinute	integer	240	Politeness throttle.
proxyConfiguration	object	no proxy	Enable Apify Proxy only if you get blocked.

Output

One dataset item per page:

{
    "url": "https://docs.apify.com/actors/publishing/publish",
    "canonicalUrl": "https://docs.apify.com/actors/publishing/publish",
    "title": "Publish your Actor",
    "description": "Prepare your Actor for publication on Apify Store.",
    "lang": "en",
    "section": "publishing",
    "breadcrumbs": ["Actors", "Publish"],
    "headings": [
        { "level": 1, "text": "Publish your Actor", "id": "" },
        { "level": 2, "text": "Before you start", "id": "before-you-start" },
        { "level": 2, "text": "Make your Actor public", "id": "make-your-actor-public" }
    ],
    "markdown": "By publishing your Actor, you make it available to the public on [Apify Store](https://apify.com/store)...\n\n## Before you start\n\n...",
    "text": "By publishing your Actor, you make it available to the public on Apify Store...",
    "wordCount": 377,
    "contentHash": "259baa8935a974386f9618d03eb24a79826823baf3894c05369c734bc4f4e61e",
    "internalLinks": ["https://docs.apify.com/actors/publishing/actor-readme", "..."],
    "depth": 2,
    "crawledAt": "2026-09-22T14:14:36.858Z",
    "markdownFileKey": "page-docs.apify.com--actors--publishing--publish.md"
}

With saveMarkdownFiles on, the key-value store also gets one file per page, for example page-docs.apify.com--actors--publishing--publish.md:

---
title: 'Publish your Actor'
source_url: 'https://docs.apify.com/actors/publishing/publish'
canonical_url: 'https://docs.apify.com/actors/publishing/publish'
description: 'Prepare your Actor for publication on Apify Store.'
lang: 'en'
section: 'publishing'
breadcrumbs:
    - 'Actors'
    - 'Publish'
word_count: 377
content_hash: '259baa89...'
depth: 2
crawled_at: '2026-09-22T14:14:36.858Z'
---

# Publish your Actor

By publishing your Actor, you make it available to the public on [Apify Store](https://apify.com/store)...

INDEX.json lists every stored page (url, title, key, wordCount, contentHash, section) together with run statistics, and SUMMARY.json holds the statistics alone. Use contentHash to detect changed pages between scheduled runs.

With a State key, the named store holds DOCS_WATCH_STATE ({ pages: { <url>: { hash, title, firstSeenAt, lastChangedAt, lastSeenAt } }, updatedAt }). It is saved at the end of the run, also when the crawl fails, and when the run is aborted, so pages that were already billed are not billed again by the next run.

Pricing

Pay-per-event. You only pay for pages written to the dataset. Platform usage (compute, storage) is included.

Event	Price	When
page-scraped	$0.002 per page ($2 per 1,000 pages)	Once per page written to the dataset. Duplicates, empty pages, unchanged pages in watch mode and failed requests are never charged.
Actor start	$0.005 per run	Apify's standard start event; covers the first seconds of compute.

Examples: a 300-page docs site costs about $0.61 for the first export. In watch mode, a daily re-crawl of the same site where 5 pages changed costs about $0.015 (start + 5 pages), about $0.45 a month. Set Max total charge per run in the run options to hard-cap spend.

Verified

Local run on 2026-09-22 with apify run (Apify CLI 1.10.0, Apify SDK 3.7.2, Crawlee 3.18.1, Node 24), input above (https://docs.apify.com/actors, maxPages: 20, concurrency 4):

• 23 requests, 23 succeeded, 0 failed, 20 pages stored, 0 duplicates, 15,165 words, runtime about 6 seconds.
• 20 page-<slug>.md files + INDEX.json + SUMMARY.json written to the key-value store (224 KB total).
• Output spot-checked: front matter, headings outline, fenced code blocks with language, tables, absolute links; no sidebar, footer, "Copy for LLM" or anchor-icon noise.
• Pay-per-event path tested with ACTOR_TEST_PAY_PER_EVENT=1 ACTOR_MAX_TOTAL_CHARGE_USD=5 (local test price $1/event): the Actor stored 5-6 pages, charged exactly that many page-scraped events, and stopped gracefully with "Run budget reached (maxTotalChargeUsd)". Concurrency may let one in-flight page complete after the limit is hit, which is the SDK's documented over-charge-by-one behaviour that the platform uses to abort the run.
• Watch mode, local runs on 2026-09-25 (https://docs.apify.com/platform/actors/running, stateKey set): run 1 exported 12 pages; run 2 exported 0 (12 unchanged, nothing billed); after editing one stored hash and adding a fake page to the state, run 3 exported exactly 1 page as changed and reported the fake page as removed, while a known page outside the scope stayed in the state.
• Abort, platform runs on 2026-10-06 (build 0.3.1, https://docs.apify.com/platform, maxPages: 25, State key set): run 1 was aborted after 14 billed pages and saved those 14 to the state while aborting; run 2 with the same key skipped the 14 for free and billed only the 10 pages it had not exported before (no page billed twice).
• Sitemap / llms.txt discovery, local runs on 2026-10-16 (build 0.4): https://docs.apify.com/platform/actors/running found 3 sitemaps and 2 llms.txt files and queued the 12 in-scope pages (also after the start URL redirected to /actors/running); https://docs.astral.sh/uv/ stored 84 pages with and without discovery (a fully linked site: no extra pages, no extra cost). https://gitbook.com/docs with maxPages: 300: without discovery 235 pages stored and 71 failed (broken in-page links); with discovery 11 sitemaps and 2 llms.txt files were read and 298 pages stored, 0 failed. Platform run HWxCnX3nbu0BmbNQY (build 0.4.1): 13 pages stored, 13 page-scraped events charged.
• Unit tests: npm test (21 tests; sitemap, sitemap index, gzip, robots.txt, llms.txt parsing, scope filtering, discovery never fails the run; watch mode: classification, scope, removed pages, state merge and cap; earlier 7 tests: extraction, code blocks, tables, admonitions, links, front matter, scope globs, key-value-store keys).

Changelog

• 0.4 (2026-10-16): pages are also discovered from sitemap.xml (robots.txt, sitemap indexes, gzip) and llms.txt, so pages that no other page links to (orphans, JS-only sidebars) are no longer missed. Discovered pages obey the same scope, exclude globs, robots.txt and maxPages. SUMMARY.json lists the sitemaps and llms.txt files used. Both can be turned off.
• 0.3 (2026-10-06): the watch state is also saved when a run is aborted (pages already billed are not billed again by the next run). README rewritten by task, with a limits table.
• 0.2 (2026-09-25): watch mode (stateKey): only new and changed pages are exported and billed, removed pages listed in CHANGES.json. Fixed a concurrency race that could store one page over maxPages.
• 0.1 (2026-09-22): initial release.

Support

Open an issue on the Actor's Issues tab. Issues are answered within one business day; bug fixes ship as new builds without changing the input schema.

Made by Vellum Labs.