Doc2RAG AI - Documentation to LLM-Ready Markdown Converter

The ultimate technical documentation scraper for AI agents and RAG pipelines. Automatically crawl, clean, and convert technical documentation websites (Stripe, Supabase, Next.js, Mintlify, Docusaurus, GitBook) into pristine, LLM-ready Markdown with preserved multi-language code snippets, tables, and RAG YAML frontmatter.

---

🎯 Why Doc2RAG AI?

General-purpose web scrapers dump navigation bars, sidebars, cookie popups, and search modals into your vector database, diluting context quality and inflating token costs.

Doc2RAG AI is engineered specifically for developers building RAG pipelines, Claude Projects, and Cursor IDE context:

• 🛠️ Framework Auto-Detection: Recognizes Mintlify, Docusaurus, GitBook, Nextra, VitePress, Readme.io, and MkDocs to isolate the technical content body with 100% precision.
• 💻 Multi-Language Code Tab Preservation: Unlike naive scrapers that drop inactive tabs, Doc2RAG parses tabbed code containers and formats every language (cURL, Python, Node.js, Go) into distinct labeled Markdown fences.
• 📊 Clean Table Conversion: Formats HTML tables into clean GitHub-flavored Markdown tables.
• 🏷️ RAG Frontmatter: Injects structured YAML frontmatter (title, url, headings, word_count, estimated_tokens, crawled_at) for vector embeddings and document chunkers.
• 📦 Triple Output Delivery:
  1. Structured JSON Dataset: Per-page records with clean Markdown and metadata.
  2. documentation.zip: Downloadable ZIP folder hierarchy mirroring documentation URL subpaths.
  3. llms-full.txt: Consolidated single document with master Table of Contents, ready to drop into Claude Projects, NotebookLM, or ChatGPT.

---

⚡ Supported Documentation Engines

Framework	Detection Signature	Code Tabs Preserved
Docusaurus	.theme-doc-markdown, article	✅ All Languages
Mintlify	#content-area, .prose	✅ All Languages
GitBook	[data-testid="page.content"]	✅ All Languages
Nextra	article.nextra-content	✅ All Languages
VitePress	div.vp-doc, main	✅ All Languages
MkDocs	article.md-content__inner	✅ All Languages
Sphinx / ReadTheDocs	div[itemprop="articleBody"]	✅ All Languages
Custom / Bespoke Docs	Trafilatura NLP Extractor Fallback	✅ All Languages

---

📥 Input Example

{
  "start_urls": [
    "https://docs.stripe.com/api"
  ],
  "max_pages": 50,
  "max_depth": 4,
  "strict_path_scoping": true,
  "preserve_code_tabs": true,
  "include_tables": true,
  "generate_zip": true,
  "generate_llms_txt": true
}

---

📤 Output Sample (Markdown with YAML Frontmatter)

---
title: "Create a PaymentIntent"
url: "https://docs.stripe.com/api/payment_intents/create"
description: "Creates a PaymentIntent object to initiate payment collection."
framework: "Docusaurus"
word_count: 1420
estimated_tokens: 1890
crawled_at: "2026-09-13T12:00:00Z"
headings:
  - "Arguments"
  - "Returns"
  - "Code Examples"
---

# Create a PaymentIntent

Creates a PaymentIntent object to initiate payment collection.

### Code Examples

**Python Example:**
```python
import stripe
stripe.api_key = "sk_test_..."
stripe.PaymentIntent.create(
  amount=2000,
  currency="usd",
  automatic_payment_methods={"enabled": True},
)

Node.js Example:

const stripe = require('stripe')('sk_test_...');
const paymentIntent = await stripe.paymentIntents.create({
  amount: 2000,
  currency: 'usd',
  automatic_payment_methods: { enabled: true },
});

---

## 🚀 Quickstart: RAG Ingestion Example (LangChain)

```python
from apify_client import ApifyClient
from langchain_community.document_loaders import ApifyDatasetLoader
from langchain_text_splitters import MarkdownHeaderTextSplitter

# 1. Run Doc2RAG AI
client = ApifyClient("YOUR_APIFY_TOKEN")
actor_call = client.actor("automation_studio/doc-to-rag-markdown-converter").call(
    run_input={"start_urls": ["https://docs.stripe.com/api"], "max_pages": 50}
)

# 2. Ingest structured Markdown into LangChain
loader = ApifyDatasetLoader(
    dataset_id=actor_call["defaultDatasetId"],
    dataset_mapping_function=lambda item: Document(
        page_content=item["markdown"],
        metadata={"url": item["url"], "title": item["title"], "tokens": item["estimated_tokens"]}
    )
)
docs = loader.load()