Docs to Markdown — Documentation Crawler for RAG with a Coverage Check

Crawl a documentation site into clean Markdown for RAG, LLM and vector database ingestion — and get a per-page coverage score that proves nothing was silently dropped.

Most web-content crawlers, including the popular ones, run Mozilla Readability to decide what the "article" on a page is. Readability was built for news. On documentation portals it sometimes throws the documentation away — and it does it silently. There is no error, no warning, and a perfectly populated markdown field. You only find out weeks later, when your docs chatbot starts inventing APIs.

This Actor exists to make that failure impossible to miss, and mostly impossible in the first place.

The problem, measured

Run the standard Readability-based pipeline over the Django QuerySet API reference — one of the most-read pages in Python documentation — and this is what comes back:

This document is for an insecure version of Django that is no longer supported.
Please upgrade to a newer release!

Django
The web framework for perfectionists with deadlines.

210 characters out of 135,448. Not an error — a deprecation banner and a footer tagline, returned as if they were the page. Every page on that site returns the same 210 characters, so a whole-site crawl produces a dataset that looks full and contains nothing.

The same pipeline over the same page with this Actor returns 130,588 characters, 96% of the page, with all 195 code blocks language-tagged.

What it does differently

1. It uses the whole crawl, not one page. Navigation, sidebars, footers and cookie banners are exactly the blocks whose text is identical across every page of a site. A single-page extractor cannot see that; a crawler can. This Actor fingerprints every DOM subtree across the pages it samples and removes what repeats — which kills the nav, the "On this page" rail and the cookie notice without you writing a single CSS selector. It never removes a block containing code, a table or a heading, and if pruning would delete most of a page it backs off and says so.

2. It measures its own output. Every item carries coverageRatio (how much of the page's available content survived), extractionStrategy (which approach won), and extractionWarnings. Extraction is a fallback ladder: if the best-scoring content region comes back thin, the next strategy is tried, down to the whole page body. A page can come out imperfect, but it cannot come out empty without telling you.

3. It keeps the things RAG actually needs. Code fences carry their language (python, not bare ), because the language class is read off the DOM before anything strips it. Tables are expanded from rowspan/colspan into a dense grid so columns stay aligned, pipes inside cells are escaped, and multi-row headers collapse to readable labels. Heading levels are preserved so downstream splitters like LangChain's MarkdownHeaderTextSplitter have something to split on.

4. It is bounded by default. maxPages defaults to 25, not to infinity. There is also a maxRunSecs wall-clock stop. Scope is one rule — a URL is crawled if it sits under the start URL's path — so a crawl started at /docs/ cannot wander into /blog/ because some other option quietly widened it.

Measured against the standard Readability pipeline

Same pages, same HTML, both pipelines run locally:

Documentation site	Readability pipeline	This Actor
docs.djangoproject.com (Sphinx)	210 chars — 0.2% of page	130,588 chars — 96%
docs.python.org (Sphinx)	59,242 chars (134% — nav bleed)	41,878 chars — 95%
docs.apify.com (Docusaurus)	49,305 chars (121% — nav bleed)	38,714 chars — 95%
kubernetes.io (Hugo)	48,418 chars	40,075 chars
developer.mozilla.org	25,826 chars	37,651 chars
doc.rust-lang.org (mdBook)	17,485 chars (111% — nav bleed)	15,358 chars — 97%
Code fences carrying a language	0 of 109	331 of 340

Percentages above 100% mean the extractor returned more than the page's own text — navigation and footer duplicated into every single record, which is what then gets embedded.

Output

One dataset item per page:

url · title · markdown · text · wordCount · charCount coverageRatio · extractionStrategy · availableChars · extractionWarnings · isLowCoverage · isClientRendered codeBlocks · codeBlocksWithLanguage · tables · headings · boilerplateBlocksRemoved depth · httpStatus · domain · path · fetchedAt · responseTimeMs

A run-level CRAWL_REPORT record holds scope, page counts, mean coverage, request errors and why the crawl stopped.

Dataset views: Overview, Markdown for ingestion (url/title/markdown), Extraction quality (the audit view), and All fields.

Good for

Documentation portals built with Docusaurus, MkDocs / MkDocs-Material, Sphinx, VitePress, Starlight, mdBook, Hugo, Docsify, GitBook, Nextra, Astro and hand-rolled docs — knowledge bases, developer guides, API references, help centres. Feed the output to LangChain, LlamaIndex, Haystack, Pinecone, Qdrant, Weaviate, Chroma, or straight into a model's context.

Not for

• Pages behind a login. This Actor sends no cookies and no credentials.
• Client-rendered single-page apps. Documentation generators ship their text in the HTML, so this crawler reads them over plain HTTP — fast and cheap. A site that builds its content in the browser will come back nearly empty, and the item will be flagged isClientRendered: true rather than quietly returning a blank. Use a browser-based crawler for those.
• Chunking and embedding. Deliberately out of scope. Chunk with your own splitter, where you know your embedding model and token budget. This Actor's job is to hand that splitter clean, structurally faithful Markdown.

Tips

• Leave skipLowCoveragePages off for a first run, sort the dataset by coverageRatio, and look at the bottom. That is your extraction audit.
• Use excludeUrlPatterns for versioned docs: excluding /v1/ and /v2/ stops the same page landing in your vector store three times.
• Raise maxPages once a small run looks right. Start at 25.