# Docs To Markdown (`excellent_mustang/docs-to-markdown`) Actor

- **URL**: https://apify.com/excellent\_mustang/docs-to-markdown.md
- **Developed by:** [Gorav Agarwal](https://apify.com/excellent_mustang) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docs to Markdown — Documentation Crawler for RAG with a Coverage Check

Crawl a **documentation site** into clean **Markdown** for **RAG**, **LLM** and **vector database** ingestion — and get a per-page **coverage score** that proves nothing was silently dropped.

Most web-content crawlers, including the popular ones, run **Mozilla Readability** to decide what the "article" on a page is. Readability was built for news. On documentation portals it sometimes throws the documentation away — and it does it *silently*. There is no error, no warning, and a perfectly populated `markdown` field. You only find out weeks later, when your docs chatbot starts inventing APIs.

This Actor exists to make that failure impossible to miss, and mostly impossible in the first place.

### The problem, measured

Run the standard Readability-based pipeline over the Django QuerySet API reference — one of the most-read pages in Python documentation — and this is what comes back:

```
This document is for an insecure version of Django that is no longer supported.
Please upgrade to a newer release!

Django
The web framework for perfectionists with deadlines.
```

**210 characters out of 135,448.** Not an error — a deprecation banner and a footer tagline, returned as if they were the page. Every page on that site returns the same 210 characters, so a whole-site crawl produces a dataset that looks full and contains nothing.

The same pipeline over the same page with this Actor returns **130,588 characters, 96% of the page, with all 195 code blocks language-tagged**.

### What it does differently

**1. It uses the whole crawl, not one page.** Navigation, sidebars, footers and cookie banners are exactly the blocks whose text is identical across every page of a site. A single-page extractor cannot see that; a crawler can. This Actor fingerprints every DOM subtree across the pages it samples and removes what repeats — which kills the nav, the "On this page" rail and the cookie notice without you writing a single CSS selector. It never removes a block containing code, a table or a heading, and if pruning would delete most of a page it backs off and says so.

**2. It measures its own output.** Every item carries `coverageRatio` (how much of the page's available content survived), `extractionStrategy` (which approach won), and `extractionWarnings`. Extraction is a **fallback ladder**: if the best-scoring content region comes back thin, the next strategy is tried, down to the whole page body. A page can come out imperfect, but it cannot come out empty without telling you.

**3. It keeps the things RAG actually needs.** Code fences carry their **language** (`python, not bare `), because the language class is read off the DOM before anything strips it. Tables are expanded from `rowspan`/`colspan` into a dense grid so columns stay aligned, pipes inside cells are escaped, and multi-row headers collapse to readable labels. Heading levels are preserved so downstream splitters like LangChain's `MarkdownHeaderTextSplitter` have something to split on.

**4. It is bounded by default.** `maxPages` defaults to **25**, not to infinity. There is also a `maxRunSecs` wall-clock stop. Scope is **one rule** — a URL is crawled if it sits under the start URL's path — so a crawl started at `/docs/` cannot wander into `/blog/` because some other option quietly widened it.

### Measured against the standard Readability pipeline

Same pages, same HTML, both pipelines run locally:

| Documentation site | Readability pipeline | This Actor |
|---|---|---|
| docs.djangoproject.com (Sphinx) | 210 chars — **0.2% of page** | 130,588 chars — 96% |
| docs.python.org (Sphinx) | 59,242 chars (134% — nav bleed) | 41,878 chars — 95% |
| docs.apify.com (Docusaurus) | 49,305 chars (121% — nav bleed) | 38,714 chars — 95% |
| kubernetes.io (Hugo) | 48,418 chars | 40,075 chars |
| developer.mozilla.org | 25,826 chars | 37,651 chars |
| doc.rust-lang.org (mdBook) | 17,485 chars (111% — nav bleed) | 15,358 chars — 97% |
| **Code fences carrying a language** | **0 of 109** | **331 of 340** |

Percentages above 100% mean the extractor returned *more* than the page's own text — navigation and footer duplicated into every single record, which is what then gets embedded.

### Output

One dataset item per page:

`url` · `title` · `markdown` · `text` · `wordCount` · `charCount`
`coverageRatio` · `extractionStrategy` · `availableChars` · `extractionWarnings` · `isLowCoverage` · `isClientRendered`
`codeBlocks` · `codeBlocksWithLanguage` · `tables` · `headings` · `boilerplateBlocksRemoved`
`depth` · `httpStatus` · `domain` · `path` · `fetchedAt` · `responseTimeMs`

A run-level `CRAWL_REPORT` record holds scope, page counts, mean coverage, request errors and why the crawl stopped.

Dataset views: **Overview**, **Markdown for ingestion** (`url`/`title`/`markdown`), **Extraction quality** (the audit view), and **All fields**.

### Good for

Documentation portals built with **Docusaurus, MkDocs / MkDocs-Material, Sphinx, VitePress, Starlight, mdBook, Hugo, Docsify, GitBook, Nextra, Astro** and hand-rolled docs — knowledge bases, developer guides, API references, help centres. Feed the output to **LangChain**, **LlamaIndex**, **Haystack**, **Pinecone**, **Qdrant**, **Weaviate**, **Chroma**, or straight into a model's context.

### Not for

- **Pages behind a login.** This Actor sends no cookies and no credentials.
- **Client-rendered single-page apps.** Documentation generators ship their text in the HTML, so this crawler reads them over plain HTTP — fast and cheap. A site that builds its content in the browser will come back nearly empty, and the item will be flagged `isClientRendered: true` rather than quietly returning a blank. Use a browser-based crawler for those.
- **Chunking and embedding.** Deliberately out of scope. Chunk with your own splitter, where you know your embedding model and token budget. This Actor's job is to hand that splitter clean, structurally faithful Markdown.

### Tips

- Leave `skipLowCoveragePages` **off** for a first run, sort the dataset by `coverageRatio`, and look at the bottom. That is your extraction audit.
- Use `excludeUrlPatterns` for versioned docs: excluding `/v1/` and `/v2/` stops the same page landing in your vector store three times.
- Raise `maxPages` once a small run looks right. Start at 25.

# Actor input Schema

## `startUrls` (type: `array`):

Where to start crawling. Paste the top of the docs section you want, for example https://docs.example.com/guide. By default the crawl stays inside that path, so a run started at /guide will not wander into /blog or /pricing.

## `maxPages` (type: `integer`):

Hard ceiling on pages fetched. This is a real limit, not a hint: the crawl stops the moment it is reached. Kept deliberately low by default so an exploratory run cannot turn into a surprise bill.

## `maxCrawlDepth` (type: `integer`):

How many link hops from a start URL to follow. 0 crawls only the start URLs themselves.

## `crawlWholeDomain` (type: `boolean`):

Ignore the start URL's path and allow any page on the same domain. Leave this off to keep a docs crawl inside the docs.

## `urlPrefixes` (type: `array`):

Override the automatic scope with an explicit list of prefixes. A page is crawled only if its URL starts with one of them.

## `excludeUrlPatterns` (type: `array`):

Regular expressions. Any URL matching one of them is skipped. Useful for changelogs, tag pages, or the /v1/ copy of versioned docs you do not want duplicated in your vector database.

## `useSitemap` (type: `boolean`):

Read /sitemap.xml and queue every in-scope URL it lists, in addition to following links. Sitemap URLs still obey the scope prefixes and the max pages limit.

## `minCoverageRatio` (type: `number`):

Coverage is the share of a page's available content text that survived extraction. Below this value the page is flagged with a warning, and the extractor retries with a more permissive strategy before giving up. Raise it to be stricter about partial extractions.

## `skipLowCoveragePages` (type: `boolean`):

Leave off to keep every page with its warning attached, which is usually what you want while you are still tuning. Turn on to keep the dataset clean for a production ingest.

## `includeHtml` (type: `boolean`):

Add the original HTML of each page to the output. Useful for auditing a page the coverage check flagged. Makes the dataset much larger.

## `concurrency` (type: `integer`):

How many pages to fetch in parallel. Lower this if the documentation site rate-limits you.

## `requestTimeoutSecs` (type: `integer`):

Seconds to wait for a single page before giving up on it and moving on.

## `maxRunSecs` (type: `integer`):

Stop crawling after this many seconds and write out whatever has been extracted so far. 0 means no limit beyond the platform run timeout. A second guardrail against a crawl that never ends.

## `userAgent` (type: `string`):

Override the browser User-Agent header sent with every request.

## `proxyConfiguration` (type: `object`):

Optional. Route requests through Apify Proxy. Most public documentation sites do not need it.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 25,
  "maxCrawlDepth": 3,
  "crawlWholeDomain": false,
  "urlPrefixes": [],
  "excludeUrlPatterns": [
    "/blog/",
    "/changelog/"
  ],
  "useSitemap": false,
  "minCoverageRatio": 0.25,
  "skipLowCoveragePages": false,
  "includeHtml": false,
  "concurrency": 8,
  "requestTimeoutSecs": 25,
  "maxRunSecs": 0,
  "userAgent": "",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

Every extracted page with its Markdown, coverage ratio and structure counts.

## `markdownForRag` (type: `string`):

Just url, title and markdown - the shape most RAG loaders want.

## `qualityReview` (type: `string`):

Coverage ratio, strategy and warnings per page, so you can spot pages that need attention before you embed them.

## `crawlReport` (type: `string`):

Run-level summary: scope, page counts, mean coverage, request errors and why the crawl stopped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy"
        }
    ],
    "excludeUrlPatterns": [
        "/blog/",
        "/changelog/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("excellent_mustang/docs-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy" }],
    "excludeUrlPatterns": [
        "/blog/",
        "/changelog/",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("excellent_mustang/docs-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "excludeUrlPatterns": [
    "/blog/",
    "/changelog/"
  ]
}' |
apify call excellent_mustang/docs-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,excellent_mustang/docs-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VhUQKDsyGne2AY8zB/builds/Fa2K3blTtfhDmsj9o/openapi.json