Pricing

Pay per usage

rag-docs-scraper

Extract clean, RAG-optimized Markdown from any technical documentation. Built for LLMs and AI agents. No noise, just high-fidelity data.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Hastin S.

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

AI Documentation & RAG Scraper 🤖📄

The AI Documentation & RAG Scraper is a high-performance tool designed to transform messy technical documentation into clean, structured Markdown. It is specifically optimized for RAG (Retrieval-Augmented Generation) pipelines, LLM fine-tuning, and AI agents.

Stop feeding your AI noisy HTML. Get the clean text you need, instantly.

✨ Key Features

Markdown Optimized: Automatically converts HTML to clean Markdown while preserving headers, code blocks, and tables.
Noise Removal: Smartly identifies and strips out navbars, footers, sidebars, and cookie banners to focus only on the content.
Modern Web Support: Powered by Playwright, it easily handles JavaScript-heavy documentation sites (React, Docusaurus, GitBook, Next.js).
Recursive Crawling: Provide a homepage, and the scraper will automatically follow internal links to map out the entire documentation set.
AI-Agent Ready: Output is structured perfectly for Vector Databases (Pinecone, Weaviate) or direct upload to ChatGPT/Claude.

🚀 How to Use

Input URLs: Enter the starting URL of the documentation you want to scrape (e.g., https://docs.apify.com/).
Set Page Limit: Define how many pages you want to crawl to stay within your budget.
Run & Download: Start the Actor and download your results in JSON, CSV, or Excel.

🛠️ Input Configuration

Field	Type	Description
Start URLs	Array	The entry points for the crawl. Supports multiple URLs.
Max Pages	Integer	The maximum number of pages to crawl (default: 50).
Proxy	Object	Uses Apify Proxy to ensure high success rates and avoid rate limits.

📊 Sample Output

{
  "url": "[https://crawlee.dev/docs/quick-start](https://crawlee.dev/docs/quick-start)",
  "title": "Quick Start | Crawlee",
  "markdown": "# Quick Start\n\nInstall Crawlee using npm...\n\n```bash\nnpm install crawlee playwright\n```",
  "scrapedAt": "2026-05-07T12:00:00Z"
}

Docs-to-RAG Crawler

automation-lab/docs-rag-crawler

Crawl documentation sites (ReadTheDocs, GitBook, Docusaurus, Mintlify) and output chunked markdown ready for RAG/vector DB ingestion. Splits by heading hierarchy, strips nav/sidebar chrome.

Stas Persiianenko

Documentation Crawler for RAG

liquid_bark/docs-crawler-for-rag

Specialized crawler for developer documentation sites. Detects frameworks (Docusaurus, GitBook, ReadTheDocs, MkDocs, Sphinx), extracts clean content, and outputs semantically chunked Markdown optimized for RAG pipelines.

Izz

Docs to Markdown + AI Embeddings → Vector DB Crawler

badruddeen/docs-to-markdown-ai-embeddings---vector-db-crawler

Turn any documentation site into clean Markdown, intelligently chunked content with embeddings (Azure/OpenAI), and directly upsert into MongoDB Atlas, Pinecone, Weaviate, Qdrant, or Milvus — ready for RAG, AI assistants, and semantic search in minutes.

Badruddeen Naseem

5.0

(1)

Docs Markdown Rag Ready Crawler

devwithbobby/docs-markdown-rag-ready-crawler

Turn any documentation site or website into clean, structured markdown—ready for RAG, embeddings, and AI agents.

Dev with Bobby

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

Dmitry Goncharov

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

nezha

5.0

(1)

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!

ParseForge

PDF Extractor: Structured Text + Metadata

aitoolbreakdown/atb-pdf-extractor

Point it at one or many PDF URLs. Get clean structured JSON back: full text, per-page text, title, author, page count, and word count. Ready for RAG, search, or doc automation.

AI Tool Breakdown

Llm Ready Documentation Scraper

direct_duty/llm-ready-documentation-scraper

Developers and AI agents need to read documentation (e.g. Stripe Docs, Next.js Docs), but standard scrapers return noisy HTML that includes: navigation bars headers / footers ads / cookie banners This Actor must return pure content-only Markdown, suitable for vectorization and semantic search.

Sean

RAG Web Browser — AI Search & Extract (10x Lighter)

tugelbay/rag-web-browser

RAG web browser for AI agents. Search Google, scrape top results, return clean Markdown. One API call replaces Tavily + scraper combo. 256MB memory, 10x lighter than alternatives. Works with LangChain, Claude MCP, OpenAI.