Docs Site to Markdown Corpus

Turn any documentation site into a clean, LLM-ready Markdown corpus in one run. Point the Actor at a docs URL and get one record per page with GitHub-flavoured Markdown, a headings outline, breadcrumbs, canonical URL, word count and a content hash, plus an optional .md file per page (with YAML front matter) ready to drop into Obsidian, Claude Code, an LLM wiki or a RAG index.

It is built for the things generic website crawlers get wrong on docs sites:

• Strips the chrome. Sidebars, top nav, "Edit this page", "Copy for LLM" buttons, heading anchor icons, breadcrumbs, pagination and cookie banners are removed before conversion. Only the article body is kept.
• Keeps what matters. Fenced code blocks with language hints (Docusaurus / Shiki / Prism line markup handled), tables, admonitions (as blockquotes), absolute links and images.
• Knows docs frameworks. Auto-detects the content container for Docusaurus, MkDocs Material, Sphinx / ReadTheDocs, VitePress, Nextra, Mintlify, GitBook and generic article / main layouts. Or pass your own CSS selector.
• Stays in scope. By default it only follows links on the same host under the same path prefix as your start URL (https://docs.example.com/guide/ never wanders into /blog/). Redirected start URLs extend the scope automatically.
• Deduplicates. Canonical-URL aliases and byte-identical pages (/ vs /index.html) are stored and billed once.
• Watch mode for RAG sync. Give it a stateKey and schedule it: every run re-crawls the site but returns and bills only new and changed pages, and lists pages that disappeared. Unchanged pages cost nothing. Keeps a vector index or LLM wiki in sync with the live docs.
• Respects robots.txt by default and throttles per run.
• No browser. Runs on plain HTTP + Cheerio, so it is fast and cheap: about 20 pages in 6 seconds on 256 MB.

Who is it for

• AI agents and MCP clients that need a whole product's docs as context (call it from the Apify MCP server, then read the dataset).
• RAG / vector-index pipelines that want section-aware Markdown instead of raw HTML.
• Knowledge bases: Obsidian vaults, Claude Code wikis, Notion imports, internal docs mirrors.
• Docs teams auditing their own site (word counts, headings outline, empty pages, duplicates).

Input

Field	Type	Default	Notes
startUrls	array	required	Docs pages to start from. Scope = same host + same path prefix.
maxPages	integer	200	Hard cap on stored pages. Each stored page is one page-scraped event, so this caps your cost.
maxCrawlDepth	integer	20	Link hops from the start URLs. 0 = start URLs only.
stateKey	string	""	Watch mode. Name of a key-value store that remembers page hashes, e.g. acme-docs-watch. Empty = one-off export.
includeUrlGlobs	string[]	[]	Override the auto scope, e.g. https://docs.example.com/**.
excludeUrlGlobs	string[]	[]	Never follow these. Binary files, feeds and search pages are always excluded.
contentSelector	string	""	CSS selector of the main content. Empty = auto-detect.
removeSelectors	string[]	[]	Extra selectors to strip (on top of the built-in noise list).
saveMarkdownFiles	boolean	true	Also write page-<slug>.md files and an INDEX.json manifest to the key-value store.
includeHtml	boolean	false	Add cleaned content HTML to each item.
respectRobotsTxt	boolean	true	Skip URLs disallowed by robots.txt.
maxConcurrency	integer	8	Parallel requests.
maxRequestsPerMinute	integer	240	Politeness throttle.
proxyConfiguration	object	no proxy	Enable Apify Proxy only if you get blocked.

Example input:

{
    "startUrls": [{ "url": "https://docs.apify.com/actors" }],
    "maxPages": 20,
    "saveMarkdownFiles": true
}

Output

One dataset item per page:

{
    "url": "https://docs.apify.com/actors/publishing/publish",
    "canonicalUrl": "https://docs.apify.com/actors/publishing/publish",
    "title": "Publish your Actor",
    "description": "Prepare your Actor for publication on Apify Store.",
    "lang": "en",
    "section": "publishing",
    "breadcrumbs": ["Actors", "Publish"],
    "headings": [
        { "level": 1, "text": "Publish your Actor", "id": "" },
        { "level": 2, "text": "Before you start", "id": "before-you-start" },
        { "level": 2, "text": "Make your Actor public", "id": "make-your-actor-public" }
    ],
    "markdown": "By publishing your Actor, you make it available to the public on [Apify Store](https://apify.com/store)...\n\n## Before you start\n\n...",
    "text": "By publishing your Actor, you make it available to the public on Apify Store...",
    "wordCount": 377,
    "contentHash": "259baa8935a974386f9618d03eb24a79826823baf3894c05369c734bc4f4e61e",
    "internalLinks": ["https://docs.apify.com/actors/publishing/actor-readme", "..."],
    "depth": 2,
    "crawledAt": "2026-09-22T14:14:36.858Z",
    "markdownFileKey": "page-docs.apify.com--actors--publishing--publish.md"
}

With saveMarkdownFiles on, the key-value store also gets one file per page, for example page-docs.apify.com--actors--publishing--publish.md:

---
title: "Publish your Actor"
source_url: "https://docs.apify.com/actors/publishing/publish"
canonical_url: "https://docs.apify.com/actors/publishing/publish"
description: "Prepare your Actor for publication on Apify Store."
lang: "en"
section: "publishing"
breadcrumbs:
  - "Actors"
  - "Publish"
word_count: 377
content_hash: "259baa89..."
depth: 2
crawled_at: "2026-09-22T14:14:36.858Z"
---

# Publish your Actor

By publishing your Actor, you make it available to the public on [Apify Store](https://apify.com/store)...

INDEX.json lists every stored page (url, title, key, wordCount, contentHash, section) together with run statistics, and SUMMARY.json holds the statistics alone. Use contentHash to detect changed pages between scheduled runs.

Watch mode: keep a RAG index in sync with the docs

A docs site changes a few pages a week. Re-exporting all 300 pages every day to find those few is slow and wasteful. Watch mode does the comparison for you:

1. Set stateKey to a name of your choice (one per site), e.g. acme-docs-watch, and set maxPages above the size of the site.
2. Run once. Every page is exported with "changeType": "new" and its hash is saved in a named key-value store with that name.
3. Schedule the same input (daily or weekly) in Apify Schedules. Each run re-crawls the site and:
   • exports and bills a page only when it is new or its Markdown changed ("changeType": "changed", plus previousContentHash);
   • skips unchanged pages for free;
   • writes CHANGES.json to the run's key-value store with the URLs that are new, changed and removed.

{
    "stateStore": "apify-docs-watch",
    "crawlComplete": true,
    "new": [],
    "changed": ["https://docs.apify.com/actors/running"],
    "removed": ["https://docs.apify.com/actors/running/old-page"],
    "unchangedCount": 11,
    "crawledAt": "2026-09-25T04:03:38.439Z"
}

Upsert the dataset items into your index by url and delete the removed URLs. A page counts as removed when it was exported before, lies inside the current scope and was not reached by this run. Removed pages are reported only after a complete crawl: if the run stops early because of maxPages or the run budget (crawlComplete: false), nothing is reported as removed, so a partial crawl never deletes pages from your index.

Pricing

This Actor uses pay-per-event pricing. You only pay for what is stored:

Event	Price	When
page-scraped	$0.002 per page ($2 per 1,000 pages)	Once per page written to the dataset. Duplicates, empty pages and failed requests are never charged.
Actor start	$0.005 per run	Apify's standard start event; covers the first seconds of compute.

A 300-page docs site therefore costs about $0.61 for the first export. In watch mode, a daily re-crawl of the same site where 5 pages changed costs about $0.015 (start + 5 pages). Set Max total charge per run in the run options to hard-cap spend; the Actor stops gracefully before exceeding it. Platform usage (compute, storage) is included in the event prices.

Tips

• Big sites: start with maxPages: 50 to check the content selector, then raise it.
• Versioned docs (/v1/, /latest/): add the old versions to excludeUrlGlobs, e.g. https://docs.example.com/v1/**.
• Wrong content picked up: pass contentSelector, e.g. .theme-doc-markdown or #main-content, and add stray elements to removeSelectors.
• JavaScript-only docs: this Actor fetches HTML over HTTP and does not run JavaScript. Most docs frameworks server-render their pages; for pure client-side apps use a browser-based crawler.
• Scheduling: use watch mode (stateKey) instead of comparing contentHash yourself.

Limitations

Stated up front so you can decide before running:

• No JavaScript rendering. Pages that only render in the browser (client-side SPAs without server-side HTML) come back empty and are skipped, not billed.
• Login-protected docs are not supported. There is no cookie or credential input.
• robots.txt is honoured by default. Disallowed paths are never fetched, even if they are linked.
• Rate limits are the target site's. Default throttle is 240 requests per minute with 8 parallel requests; on HTTP 429 or 5xx the request is retried twice, then counted in pagesFailed and not billed. Lower maxConcurrency for small sites.
• Change detection is by Markdown content. Pages that print a timestamp or random token inside the article body will show up as changed on every run; remove that element with removeSelectors.
• Removed = no longer reachable. A page that still exists but is no longer linked from inside the scope is reported as removed.
• The watch state keeps the 50,000 most recently seen pages per stateKey.

Verified

Local run on 2026-09-22 with apify run (Apify CLI 1.10.0, Apify SDK 3.7.2, Crawlee 3.18.1, Node 24), input above (https://docs.apify.com/actors, maxPages: 20, concurrency 4):

• 23 requests, 23 succeeded, 0 failed, 20 pages stored, 0 duplicates, 15,165 words, runtime about 6 seconds.
• 20 page-<slug>.md files + INDEX.json + SUMMARY.json written to the key-value store (224 KB total).
• Output spot-checked: front matter, headings outline, fenced code blocks with language, tables, absolute links; no sidebar, footer, "Copy for LLM" or anchor-icon noise.
• Pay-per-event path tested with ACTOR_TEST_PAY_PER_EVENT=1 ACTOR_MAX_TOTAL_CHARGE_USD=5 (local test price $1/event): the Actor stored 5-6 pages, charged exactly that many page-scraped events, and stopped gracefully with "Run budget reached (maxTotalChargeUsd)". Concurrency may let one in-flight page complete after the limit is hit, which is the SDK's documented over-charge-by-one behaviour that the platform uses to abort the run.
• Watch mode, local runs on 2026-09-25 (https://docs.apify.com/platform/actors/running, stateKey set): run 1 exported 12 pages; run 2 exported 0 (12 unchanged, nothing billed); after editing one stored hash and adding a fake page to the state, run 3 exported exactly 1 page as changed and reported the fake page as removed, while a known page outside the scope stayed in the state.
• Unit tests: npm test (14 tests; watch mode: classification, scope, removed pages, state merge and cap; earlier 7 tests: extraction, code blocks, tables, admonitions, links, front matter, scope globs, key-value-store keys).

Support

Open an issue on the Actor's Issues tab. Issues are answered within one business day; bug fixes ship as new builds without changing the input schema.

Made by Vellum Labs.