# Docs Site to Markdown Corpus (`vellumlabs/docs-to-markdown-corpus`) Actor

Turn any documentation site into a clean Markdown corpus: one file per page with front matter (title, URL, breadcrumbs), headings, code blocks with languages, tables and absolute links, plus an INDEX.json. Built for LLM wikis, RAG and offline docs. Pay per page.

- **URL**: https://apify.com/vellumlabs/docs-to-markdown-corpus.md
- **Developed by:** [Vellum Kasane](https://apify.com/vellumlabs) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docs Site to Markdown Corpus

Turn any documentation site into a **clean, LLM-ready Markdown corpus** in one run. Point the Actor at a docs URL and get one record per page with GitHub-flavoured Markdown, a headings outline, breadcrumbs, canonical URL, word count and a content hash, plus an optional `.md` file per page (with YAML front matter) ready to drop into Obsidian, Claude Code, an LLM wiki or a RAG index.

It is built for the things generic website crawlers get wrong on docs sites:

- **Strips the chrome.** Sidebars, top nav, "Edit this page", "Copy for LLM" buttons, heading anchor icons, breadcrumbs, pagination and cookie banners are removed before conversion. Only the article body is kept.
- **Keeps what matters.** Fenced code blocks with language hints (Docusaurus / Shiki / Prism line markup handled), tables, admonitions (as blockquotes), absolute links and images.
- **Knows docs frameworks.** Auto-detects the content container for Docusaurus, MkDocs Material, Sphinx / ReadTheDocs, VitePress, Nextra, Mintlify, GitBook and generic `article` / `main` layouts. Or pass your own CSS selector.
- **Stays in scope.** By default it only follows links on the same host under the same path prefix as your start URL (`https://docs.example.com/guide/` never wanders into `/blog/`). Redirected start URLs extend the scope automatically.
- **Deduplicates.** Canonical-URL aliases and byte-identical pages (`/` vs `/index.html`) are stored and billed once.
- **Watch mode for RAG sync.** Give it a `stateKey` and schedule it: every run re-crawls the site but returns and bills **only new and changed pages**, and lists pages that disappeared. Unchanged pages cost nothing. Keeps a vector index or LLM wiki in sync with the live docs.
- **Respects robots.txt** by default and throttles per run.
- **No browser.** Runs on plain HTTP + Cheerio, so it is fast and cheap: about 20 pages in 6 seconds on 256 MB.

### Who is it for

- **AI agents and MCP clients** that need a whole product's docs as context (call it from the [Apify MCP server](https://mcp.apify.com), then read the dataset).
- **RAG / vector-index pipelines** that want section-aware Markdown instead of raw HTML.
- **Knowledge bases**: Obsidian vaults, Claude Code wikis, Notion imports, internal docs mirrors.
- **Docs teams** auditing their own site (word counts, headings outline, empty pages, duplicates).

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `startUrls` | array | required | Docs pages to start from. Scope = same host + same path prefix. |
| `maxPages` | integer | 200 | Hard cap on stored pages. Each stored page is one `page-scraped` event, so this caps your cost. |
| `maxCrawlDepth` | integer | 20 | Link hops from the start URLs. `0` = start URLs only. |
| `stateKey` | string | `""` | Watch mode. Name of a key-value store that remembers page hashes, e.g. `acme-docs-watch`. Empty = one-off export. |
| `includeUrlGlobs` | string\[] | `[]` | Override the auto scope, e.g. `https://docs.example.com/**`. |
| `excludeUrlGlobs` | string\[] | `[]` | Never follow these. Binary files, feeds and search pages are always excluded. |
| `contentSelector` | string | `""` | CSS selector of the main content. Empty = auto-detect. |
| `removeSelectors` | string\[] | `[]` | Extra selectors to strip (on top of the built-in noise list). |
| `saveMarkdownFiles` | boolean | `true` | Also write `page-<slug>.md` files and an `INDEX.json` manifest to the key-value store. |
| `includeHtml` | boolean | `false` | Add cleaned content HTML to each item. |
| `respectRobotsTxt` | boolean | `true` | Skip URLs disallowed by robots.txt. |
| `maxConcurrency` | integer | 8 | Parallel requests. |
| `maxRequestsPerMinute` | integer | 240 | Politeness throttle. |
| `proxyConfiguration` | object | no proxy | Enable Apify Proxy only if you get blocked. |

Example input:

```json
{
    "startUrls": [{ "url": "https://docs.apify.com/actors" }],
    "maxPages": 20,
    "saveMarkdownFiles": true
}
```

### Output

One dataset item per page:

```json
{
    "url": "https://docs.apify.com/actors/publishing/publish",
    "canonicalUrl": "https://docs.apify.com/actors/publishing/publish",
    "title": "Publish your Actor",
    "description": "Prepare your Actor for publication on Apify Store.",
    "lang": "en",
    "section": "publishing",
    "breadcrumbs": ["Actors", "Publish"],
    "headings": [
        { "level": 1, "text": "Publish your Actor", "id": "" },
        { "level": 2, "text": "Before you start", "id": "before-you-start" },
        { "level": 2, "text": "Make your Actor public", "id": "make-your-actor-public" }
    ],
    "markdown": "By publishing your Actor, you make it available to the public on [Apify Store](https://apify.com/store)...\n\n## Before you start\n\n...",
    "text": "By publishing your Actor, you make it available to the public on Apify Store...",
    "wordCount": 377,
    "contentHash": "259baa8935a974386f9618d03eb24a79826823baf3894c05369c734bc4f4e61e",
    "internalLinks": ["https://docs.apify.com/actors/publishing/actor-readme", "..."],
    "depth": 2,
    "crawledAt": "2026-09-22T14:14:36.858Z",
    "markdownFileKey": "page-docs.apify.com--actors--publishing--publish.md"
}
```

With `saveMarkdownFiles` on, the key-value store also gets one file per page, for example `page-docs.apify.com--actors--publishing--publish.md`:

```markdown
---
title: "Publish your Actor"
source_url: "https://docs.apify.com/actors/publishing/publish"
canonical_url: "https://docs.apify.com/actors/publishing/publish"
description: "Prepare your Actor for publication on Apify Store."
lang: "en"
section: "publishing"
breadcrumbs:
  - "Actors"
  - "Publish"
word_count: 377
content_hash: "259baa89..."
depth: 2
crawled_at: "2026-09-22T14:14:36.858Z"
---

## Publish your Actor

By publishing your Actor, you make it available to the public on [Apify Store](https://apify.com/store)...
```

`INDEX.json` lists every stored page (`url`, `title`, `key`, `wordCount`, `contentHash`, `section`) together with run statistics, and `SUMMARY.json` holds the statistics alone. Use `contentHash` to detect changed pages between scheduled runs.

### Watch mode: keep a RAG index in sync with the docs

A docs site changes a few pages a week. Re-exporting all 300 pages every day to find those few is slow and wasteful. Watch mode does the comparison for you:

1. Set `stateKey` to a name of your choice (one per site), e.g. `acme-docs-watch`, and set `maxPages` above the size of the site.
2. Run once. Every page is exported with `"changeType": "new"` and its hash is saved in a named key-value store with that name.
3. Schedule the same input (daily or weekly) in Apify **Schedules**. Each run re-crawls the site and:
   - exports and bills a page only when it is **new** or its Markdown **changed** (`"changeType": "changed"`, plus `previousContentHash`);
   - skips unchanged pages **for free**;
   - writes `CHANGES.json` to the run's key-value store with the URLs that are `new`, `changed` and `removed`.

```json
{
    "stateStore": "apify-docs-watch",
    "crawlComplete": true,
    "new": [],
    "changed": ["https://docs.apify.com/actors/running"],
    "removed": ["https://docs.apify.com/actors/running/old-page"],
    "unchangedCount": 11,
    "crawledAt": "2026-09-25T04:03:38.439Z"
}
```

Upsert the dataset items into your index by `url` and delete the `removed` URLs. A page counts as **removed** when it was exported before, lies inside the current scope and was not reached by this run. Removed pages are reported **only after a complete crawl**: if the run stops early because of `maxPages` or the run budget (`crawlComplete: false`), nothing is reported as removed, so a partial crawl never deletes pages from your index.

### Pricing

This Actor uses **pay-per-event** pricing. You only pay for what is stored:

| Event | Price | When |
|---|---|---|
| `page-scraped` | **$0.002 per page** ($2 per 1,000 pages) | Once per page written to the dataset. Duplicates, empty pages and failed requests are never charged. |
| Actor start | $0.005 per run | Apify's standard start event; covers the first seconds of compute. |

A 300-page docs site therefore costs about **$0.61** for the first export. In watch mode, a daily re-crawl of the same site where 5 pages changed costs about **$0.015** (start + 5 pages). Set **Max total charge per run** in the run options to hard-cap spend; the Actor stops gracefully before exceeding it. Platform usage (compute, storage) is included in the event prices.

### Tips

- **Big sites**: start with `maxPages: 50` to check the content selector, then raise it.
- **Versioned docs** (`/v1/`, `/latest/`): add the old versions to `excludeUrlGlobs`, e.g. `https://docs.example.com/v1/**`.
- **Wrong content picked up**: pass `contentSelector`, e.g. `.theme-doc-markdown` or `#main-content`, and add stray elements to `removeSelectors`.
- **JavaScript-only docs**: this Actor fetches HTML over HTTP and does not run JavaScript. Most docs frameworks server-render their pages; for pure client-side apps use a browser-based crawler.
- **Scheduling**: use watch mode (`stateKey`) instead of comparing `contentHash` yourself.

### Limitations

Stated up front so you can decide before running:

- **No JavaScript rendering.** Pages that only render in the browser (client-side SPAs without server-side HTML) come back empty and are skipped, not billed.
- **Login-protected docs are not supported.** There is no cookie or credential input.
- **robots.txt is honoured by default.** Disallowed paths are never fetched, even if they are linked.
- **Rate limits are the target site's.** Default throttle is 240 requests per minute with 8 parallel requests; on HTTP 429 or 5xx the request is retried twice, then counted in `pagesFailed` and not billed. Lower `maxConcurrency` for small sites.
- **Change detection is by Markdown content.** Pages that print a timestamp or random token inside the article body will show up as `changed` on every run; remove that element with `removeSelectors`.
- **Removed = no longer reachable.** A page that still exists but is no longer linked from inside the scope is reported as removed.
- **The watch state keeps the 50,000 most recently seen pages** per `stateKey`.

### Verified

Local run on 2026-09-22 with `apify run` (Apify CLI 1.10.0, Apify SDK 3.7.2, Crawlee 3.18.1, Node 24), input above (`https://docs.apify.com/actors`, `maxPages: 20`, concurrency 4):

- 23 requests, 23 succeeded, 0 failed, **20 pages stored**, 0 duplicates, 15,165 words, runtime about 6 seconds.
- 20 `page-<slug>.md` files + `INDEX.json` + `SUMMARY.json` written to the key-value store (224 KB total).
- Output spot-checked: front matter, headings outline, fenced code blocks with language, tables, absolute links; no sidebar, footer, "Copy for LLM" or anchor-icon noise.
- Pay-per-event path tested with `ACTOR_TEST_PAY_PER_EVENT=1 ACTOR_MAX_TOTAL_CHARGE_USD=5` (local test price $1/event): the Actor stored 5-6 pages, charged exactly that many `page-scraped` events, and stopped gracefully with "Run budget reached (maxTotalChargeUsd)". Concurrency may let one in-flight page complete after the limit is hit, which is the SDK's documented over-charge-by-one behaviour that the platform uses to abort the run.
- Watch mode, local runs on 2026-09-25 (`https://docs.apify.com/platform/actors/running`, `stateKey` set): run 1 exported 12 pages; run 2 exported **0** (12 unchanged, nothing billed); after editing one stored hash and adding a fake page to the state, run 3 exported exactly 1 page as `changed` and reported the fake page as `removed`, while a known page outside the scope stayed in the state.
- Unit tests: `npm test` (14 tests; watch mode: classification, scope, removed pages, state merge and cap; earlier 7 tests: extraction, code blocks, tables, admonitions, links, front matter, scope globs, key-value-store keys).

### Support

Open an issue on the Actor's **Issues** tab. Issues are answered within one business day; bug fixes ship as new builds without changing the input schema.

Made by Vellum Labs.

# Actor input Schema

## `startUrls` (type: `array`):

Documentation pages to start from. By default the crawler stays on the same host and under the same path prefix as each start URL (e.g. `https://docs.example.com/guide/` only crawls `/guide/**`). Use **Include URL globs** to widen or narrow the scope.

## `maxPages` (type: `integer`):

Hard cap on the number of pages crawled in one run. Each stored page is one `page-scraped` billing event, so this also caps your cost.

## `maxCrawlDepth` (type: `integer`):

How many link hops away from the start URLs the crawler may go. `0` = only the start URLs.

## `stateKey` (type: `string`):

Name of a key-value store that remembers the content hash of every page already exported, e.g. `apify-docs-watch`. With a state key, each run re-crawls the site but returns and bills **only pages that are new or whose content changed** since the previous run with the same key; unchanged pages are skipped for free. Pages that disappeared are listed in `CHANGES.json` (only after a complete crawl, i.e. not cut by Max pages or the budget). Leave empty for a normal one-off export. Use one key per site / per scope, and set Max pages above the site's size.

## `includeUrlGlobs` (type: `array`):

Only follow links matching at least one of these globs (e.g. `https://docs.example.com/guide/**`). Leave empty to derive one glob per start URL automatically (same host, same path prefix).

## `excludeUrlGlobs` (type: `array`):

Never follow links matching these globs. Binary files, feeds, search and edit pages are always excluded.

## `contentSelector` (type: `string`):

CSS selector of the main content element. Leave empty to auto-detect (`article`, `main`, `[role=main]`, common docs-framework wrappers such as Docusaurus, MkDocs, GitBook, ReadTheDocs, VitePress, Mintlify, Sphinx, Nextra).

## `removeSelectors` (type: `array`):

Elements removed from the content before conversion. Added on top of the built-in list (navigation, sidebars, footers, edit links, copy buttons, cookie banners, table-of-contents, breadcrumbs, scripts, styles).

## `saveMarkdownFiles` (type: `boolean`):

When enabled, each page is additionally written to the run's key-value store as `page-<slug>.md` with YAML front matter, and an `INDEX.json` manifest is written at the end. Handy for dropping straight into Obsidian, Claude Code or an LLM wiki.

## `includeHtml` (type: `boolean`):

Adds the cleaned content HTML to each dataset item. Increases dataset size.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by the target site's robots.txt. Strongly recommended; leave on unless you own the site.

## `maxConcurrency` (type: `integer`):

Maximum parallel requests. Lower it for small or rate-limited sites.

## `maxRequestsPerMinute` (type: `integer`):

Politeness throttle per run.

## `proxyConfiguration` (type: `object`):

Proxy settings. Most public docs sites need no proxy; enable Apify Proxy only if you get blocked.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/actors"
    }
  ],
  "maxPages": 20,
  "maxCrawlDepth": 20,
  "stateKey": "",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "contentSelector": "",
  "removeSelectors": [],
  "saveMarkdownFiles": true,
  "includeHtml": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 8,
  "maxRequestsPerMinute": 240,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `markdownFiles` (type: `string`):

No description

## `index` (type: `string`):

No description

## `changes` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/actors"
        }
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("vellumlabs/docs-to-markdown-corpus").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/actors" }],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("vellumlabs/docs-to-markdown-corpus").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/actors"
    }
  ],
  "maxPages": 20
}' |
apify call vellumlabs/docs-to-markdown-corpus --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vellumlabs/docs-to-markdown-corpus"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8o2nFLmVJYpv01owo/builds/JrpL3xTsLGbxeg8YW/openapi.json