Pricing

from $0.70 / 1,000 item scrapeds

Try for free

Go to Apify Store

Documentation to Clean LLM Markdown Crawler

Try for free

Crawl developer documentation sites (Docusaurus, GitBook, Mintlify, ReadMe) and extract clean, token-efficient GitHub-Flavored Markdown for Cursor, Copilot, and RAG.

Pricing

from $0.70 / 1,000 item scrapeds

Rating

0.0

(0)

Developer

LiteZero

Actor stats

Bookmarked

Total users

Monthly active users

10 days ago

Last modified

📚 Documentation to Clean LLM Markdown Crawler

Crawl developer documentation sites (Docusaurus, GitBook, Mintlify, ReadMe, Nextra) and convert complex web pages into token-efficient GitHub-Flavored Markdown. Automatically strips navigation sidebars, headers, and footers while preserving code snippets, headers, and formatting for Cursor, Windsurf, Claude Projects, and RAG vector databases.

🥊 Why This Actor Beats the Competition

Feature	This Actor (Pay-Per-Result)	Heavy Browser Crawlers (Playwright)	Standard HTML Scrapers
Speed	⚡ Fast HTTP Cheerio parsing (<1s/page)	3–5 seconds per page	Fast, but returns raw noisy HTML
Token Efficiency	🎯 Strips 85%+ of boilerplate chrome	Ingests navbars, footers & cookie banners	Ingests scripts, SVGs & layout markup
Code Block Integrity	✅ Preserves language & code syntax	Often merges code into plain text	Escapes HTML entities improperly
Pricing	💰 $1.00 / 1,000 doc pages (PPR)	High compute memory costs	Unclear per-result billing

🛠️ Step-by-Step: How to Use

Method 1: Apify Web Console

Enter the root documentation link in Documentation Start URLs (e.g. https://docs.apify.com/academy/apify-scrapers).
Set Max Pages to Crawl (e.g. 5).
Click Start and download the clean .md files or JSON dataset.

Method 2: Python SDK

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("litezero/documentation-markdown-crawler").call(run_input={
    "startUrls": ["https://docs.apify.com/academy/apify-scrapers"],
    "maxPages": 5,
    "stripNavigation": True
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Title: {item['title']} | Tokens: ~{item['tokenEstimate']}")

📥 Input Parameters

Parameter	Type	Default	Description
`startUrls`	Array	`["https://docs.apify.com/academy/apify-scrapers"]`	Documentation URLs to crawl and convert.
`maxPages`	Integer	`3`	Maximum documentation pages to crawl.
`stripNavigation`	Boolean	`true`	Automatically strip sidebars, navigation, and headers.

📤 Output Sample JSON

{
  "url": "https://docs.apify.com/academy/apify-scrapers",
  "title": "Apify scrapers | Apify Documentation",
  "markdown": "# Apify scrapers\n\n> Source: https://docs.apify.com/academy/apify-scrapers\n\n## Overview\n\nLearn how to configure Apify scrapers...",
  "wordCount": 850,
  "tokenEstimate": 1105,
  "headings": ["Overview", "Configuration", "Data Delivery"],
  "codeBlocksCount": 4,
  "scrapedAt": "2026-10-03T09:20:00.000Z"
}

💡 Top Commercial Use Cases

AI Coding Assistant Ingestion: Feed custom API documentation directly into Cursor .cursorrules or Windsurf context.
Enterprise RAG Knowledge Bases: Ingest partner and supplier technical docs into Pinecone, Qdrant, or Weaviate.
Offline Technical Archival: Convert sprawling web documentation portals into portable Markdown repositories.\n

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Hounderd

Website Content to Markdown for LLM Training

easyapi/website-content-to-markdown-for-llm-training

🚀 Transform web content into clean, LLM-ready Markdown! 📘 Scrape multiple pages, extract main content, and convert to Markdown format. Perfect for AI researchers, data scientists, and LLM developers. Fast, efficient, and customizable. Supercharge your AI training data today! 🌐📝🧠

EasyApi

338

5.0

(2)

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

Dmitry Goncharov

Website Content Crawler — Markdown for LLM & RAG

brilliant_gum/web-content-crawler

Deep-crawl websites and extract LLM-ready Markdown with OG tags, JSON-LD, author, dates, token estimates, native RAG chunking, language filtering, content-hash dedup, and per-page error reporting. Enforced timeouts. Zero silent failures.

Yuliia Kulakova

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Top Api

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Ryan Clinton

Universal Web to Markdown (Bulk & AI-Ready)

lentic_october/web-to-markdown-converter

Bulk convert any website URLs to clean Markdown for AI & LLMs. Universal scraper that removes ads, scripts, and clutter. Optimized for RAG, ChatGPT, Claude, and LangChain. Fast, async, and API-ready.

kalthireddy Abhishek

Github Repo Markdown Scraper

louisdeconinck/github-repo-markdown-scraper

Transform GitHub repositories into a single, comprehensive markdown document effortlessly. Our tool streamlines analysis and processing, offering configurable file size limits, pattern filtering, and batch processing. Perfect for LLM AI prompts, it handles large repositories with ease.