Pricing

from $3.00 / 1,000 results

Try for free

Go to Apify Store

Documentation Site Scraper

Try for free

Scrape any documentation site to clean markdown. Works on Docusaurus, Mintlify, GitBook, MkDocs, ReadTheDocs and more. Preserves code blocks for RAG and LLM training.

Pricing

from $3.00 / 1,000 results

Rating

0.0

(0)

Developer

Neil Sangwaiya

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

What you get

Field	Description
`url`	Page URL
`title`	The page's own heading, not the site name
`description`	Meta description
`markdown`	Full page content as clean markdown
`wordCount`	Words on the page
`codeBlocks`	Number of fenced code blocks
`headings`	Every heading, in order, for chunking
`site`	Source site
`scrapedAt`	ISO timestamp

Why this one is different

Code blocks survive. Syntax highlighters wrap every line in its own element, and a naive text extraction glues them into one unreadable line. Most docs scrapers return npm installcd my-appnpm run dev. This one reconstructs the lines and returns a proper fenced block with the language tag. For documentation, mangled code is worse than no code.

Titles are the page's own heading. Falling back to the <title> tag gives every page the same site-wide name, which makes a dataset useless for retrieval. This reads the real h1 first, before any markup transformation destroys it.

Discovery actually works. Sitemaps come in three shapes: a plain sitemap, an index pointing at more sitemaps, or missing entirely with the real one declared in robots.txt. Docs on a subpath often have their own separate sitemap. This handles all of it, and falls back to a scoped link crawl when there is no sitemap at all.

It scopes to what you asked for. Give it example.com/docs and you get the docs, not the blog, the pricing page and every translation. Root sitemaps list the whole site; this narrows to your path automatically.

Headings are extracted separately so you can chunk by section rather than by arbitrary character count, which is the single biggest quality lever in a RAG pipeline.

Example input

{
  "startUrl": "https://docs.apify.com",
  "maxPages": 500,
  "excludePattern": "/blog/|/changelog/"
}

Filters

Only URLs matching — a regular expression, e.g. /api/ for just the API reference
Skip URLs matching — e.g. /blog/|/es/|/fr/ to drop blogs and translations
Stay on the same host — on by default, and treats www. as the same site
Max pages — caps the run so cost is predictable

Notes

Strips navigation, sidebars, footers, breadcrumbs, cookie prompts and skip links before extraction.
Retries with backoff and respects rate limits.
Uses only public pages. No login, no private docs.

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

Dmitry Goncharov

Documentation Crawler for RAG

liquid_bark/docs-crawler-for-rag

Specialized crawler for developer documentation sites. Detects frameworks (Docusaurus, GitBook, ReadTheDocs, MkDocs, Sphinx), extracts clean content, and outputs semantically chunked Markdown optimized for RAG pipelines.

Izz

Mintlify Docs Scraper — Docs to Clean Markdown

straightforward_hydra/mintlify-docs-scraper

Extract any Mintlify / llms.txt documentation site as clean Markdown: one row per page with title, section, summary & content. For RAG & LLMs. No key.

Dev D

Docs Markdown Rag Ready Crawler

devwithbobby/docs-markdown-rag-ready-crawler

Turn any documentation site or website into clean, structured markdown—ready for RAG, embeddings, and AI agents.

Dev with Bobby

RAG-Ready Documentation Scraper

alaricus/rag-docs-markdown-scraper

Scrape documentation to framework-optimized Markdown. Features semantic chunking for LLM, vector database, and RAG pipelines. Parse XML sitemaps easily.

Alaricus

Site to Markdown — any site to clean, LLM-ready markdown

topsail/site-to-markdown

Scrape any website to clean, LLM-ready markdown — a compliant Firecrawl alternative for RAG ingestion, robots.txt always on.

Connor Teskey

Product Documentation Change Monitor scraper

funny_electrician/Korak1910

Product Documentation Change Monitor scraper: Alerts AI agents when an API or library's documentation updates.

Milton Gardener

HTML to Markdown Converter - Bulk Web Content to MD

santamaria-automations/html-to-markdown

Extract main article content from any website and convert to clean Markdown including headings, links, images, tables, and code blocks. Perfect for LLM training, AI pipelines, and documentation. Export data, run via API, schedule and monitor runs, or integrate with other tools.