Docs2RAG — Documentation-to-RAG Dataset Builder

Crawl public documentation websites and export clean Markdown, structured JSON, and RAG-ready chunks for AI agents, LLM applications, and vector databases.

---

💡 Why Docs2RAG?

Documentation websites are essential sources of truth for developers and AI agents. However, raw HTML contains navigation bars, sidebars, cookie banners, footers, search dialogs, and repetitive layout boilerplate that degrades vector retrieval and pollutes LLM context windows.

Docs2RAG solves this by:

1. Intelligently stripping layout noise (navbars, sidebars, cookie banners, footers) while keeping the core semantic article body.
2. Preserving critical technical structures: fenced code blocks with language tags, GitHub Flavored Markdown (GFM) tables, lists, and heading hierarchies.
3. Producing section-aware RAG chunks: chunked by heading paths (["Getting Started", "Installation"]) without truncating code blocks midway.
4. Providing deterministic identifiers and content hashes for incremental sync and deduplication.
5. Exporting ready-to-ingest formats: Apify Dataset, Key-Value Store Markdown files (DOC_*.md), JSON, and newline-delimited JSON (JSONL) ready for Pinecone, Qdrant, Weaviate, Chroma, Supabase, LangChain, and LlamaIndex.

---

🚀 Features

• Semantic Content Extraction: Automatic detection of <main>, <article>, #content, .docs-content, .markdown-body with customizable CSS selectors.
• Noise & Boilerplate Stripping: Removes banners, headers, footers, sidebars, cookie notices, and ads.
• High-Fidelity Markdown: Preserves ATX headings, bold/italics, links, GFM tables, and fenced code blocks with language specifiers.
• Hierarchy-Aware RAG Chunking: Chunks preserve section context and breadcrumbs, respect token boundaries, and prevent code block splitting.
• Sitemap & Deep Crawling: Discovers pages via /sitemap.xml and internal links with configurable depth and pattern filters.
• Deduplication & Incremental Hashing: Canonical URL normalization, query parameter cleaning, and SHA-256 content hashing.
• Zero Mandatory API Keys: Runs locally or in the cloud without requiring OpenAI or external embedding API keys.
• Security & SSRF Protection: Restricts schemes (http/https), blocks private IP ranges, loopback addresses (127.0.0.1, ::1), and cloud metadata services (169.254.169.254).

---

⚙️ Actor Input Configuration

Parameter	Type	Default	Description
startUrls	Array	Required	Array of entry documentation URLs (e.g. ["https://docs.docker.com/get-started/"]).
maxPages	Integer	100	Maximum number of documentation pages to crawl.
maxDepth	Integer	10	Maximum link crawl depth from start URLs.
includeUrlPatterns	Array<String>	[]	Regex or glob patterns to include (e.g. ["/docs/.*"]).
excludeUrlPatterns	Array<String>	[]	Regex or glob patterns to exclude (e.g. [".*release-notes.*"]).
sameDomainOnly	Boolean	true	Restrict crawl strictly to the same domain as start URL.
respectRobotsTxt	Boolean	true	Comply with robots.txt directives.
maxConcurrency	Integer	5	Maximum concurrent request workers.
requestTimeoutSecs	Integer	30	Request timeout per page in seconds.
chunkSize	Integer	1200	Target character size per text chunk.
chunkOverlap	Integer	200	Character overlap between consecutive chunks.
minContentLength	Integer	100	Minimum character length for a page to be processed.
includeCode	Boolean	true	Preserve code snippets and syntax highlighting.
includeImages	Boolean	false	Preserve image tags with alt text and URLs.
outputFormats	Array<String>	["MARKDOWN", "JSON", "JSONL"]	Formats to generate.
deduplicateContent	Boolean	true	Skip duplicate pages with matching content hashes.
crawlSitemap	Boolean	true	Automatically discover URLs from /sitemap.xml.
generatePageSummaries	Boolean	false	Generate extractive summaries for each page.
contentSelectors	Array<String>	[]	Optional custom CSS selectors for main content.
removeSelectors	Array<String>	[]	Optional custom CSS selectors to remove.

---

📋 Example Actor Input

{
  "startUrls": [
    { "url": "https://docs.docker.com/get-started/" }
  ],
  "maxPages": 50,
  "maxDepth": 5,
  "chunkSize": 1000,
  "chunkOverlap": 150,
  "includeCode": true,
  "outputFormats": ["MARKDOWN", "JSON", "JSONL"],
  "crawlSitemap": true,
  "deduplicateContent": true
}

---

📦 Output Formats & Schemas

1. Default Apify Dataset

Records in the dataset include both full PageRecord items and ChunkRecord items.

Sample ChunkRecord

{
  "recordType": "chunk",
  "chunkId": "guide-installation-c0-a1b2c3d4",
  "pageId": "guide-installation-9e8d7c6b5a4f3e2d",
  "url": "https://docs.example.com/guide/installation",
  "title": "Installation Guide — MyDocs",
  "section": "Prerequisites",
  "headingPath": ["Installation Guide", "Prerequisites"],
  "chunkIndex": 0,
  "content": "Before installing the SDK, ensure you have Node.js 18 or higher installed on your system.",
  "contentType": "text",
  "tokenCount": 18,
  "contentHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "metadata": {
    "sourceDomain": "docs.example.com",
    "language": "en",
    "characterCount": 89,
    "wordCount": 15,
    "crawledAt": "2026-10-10T12:00:00.000Z"
  }
}

Sample PageRecord

{
  "recordType": "page",
  "pageId": "guide-installation-9e8d7c6b5a4f3e2d",
  "url": "https://docs.example.com/guide/installation",
  "canonicalUrl": "https://docs.example.com/guide/installation",
  "title": "Installation Guide — MyDocs",
  "description": "Step by step instructions to install and configure MyDocs.",
  "sourceDomain": "docs.example.com",
  "language": "en",
  "headings": [
    { "level": 1, "text": "Installation Guide" },
    { "level": 2, "text": "Prerequisites" },
    { "level": 2, "text": "Quick Start" }
  ],
  "contentHash": "8f481c47be3c4dd2d4...",
  "crawledAt": "2026-10-10T12:00:00.000Z",
  "httpStatus": 200,
  "wordCount": 420,
  "characterCount": 2650,
  "chunkCount": 3,
  "content": "Full extracted plain text...",
  "contentMarkdown": "# Installation Guide\n\n## Prerequisites\n\nBefore installing...",
  "summary": "This document covers Installation Guide. Key sections include: Prerequisites, Quick Start."
}

2. Key-Value Store Artifacts

• DOC_{pageId}.md: Individual Markdown files with frontmatter.
• CHUNKS_JSONL: Newline-delimited JSON file of all chunks for direct vector ingestion.
• PAGES_JSONL: Newline-delimited JSON file of all pages.
• MANIFEST: Comprehensive run statistics, timings, and configuration snapshot.

---

🔌 Vector Database Integration Examples

Ingesting CHUNKS_JSONL into Vector Databases

import json
from openai import OpenAI
from pinecone import Pinecone

# 1. Initialize Clients
openai_client = OpenAI(api_key="your-openai-api-key")
pc = Pinecone(api_key="your-pinecone-api-key")
index = pc.Index("docs-knowledge-base")

# 2. Read Chunks JSONL exported from Docs2RAG
with open("CHUNKS_JSONL.jsonl", "r") as f:
    chunks = [json.loads(line) for line in f]

# 3. Generate Embeddings & Upsert in Batches
batch_size = 100
for i in range(0, len(chunks), batch_size):
    batch = chunks[i:i + batch_size]
    texts = [c["content"] for c in batch]
    
    # Generate embeddings with text-embedding-3-small
    res = openai_client.embeddings.create(input=texts, model="text-embedding-3-small")
    embeddings = [r.embedding for r in res.data]
    
    # Prepare Pinecone vectors
    vectors = [
        {
            "id": c["chunkId"],
            "values": emb,
            "metadata": {
                "url": c["url"],
                "title": c["title"],
                "section": c["section"],
                "headingPath": " > ".join(c["headingPath"]),
                "content": c["content"],
                "sourceDomain": c["metadata"]["sourceDomain"]
            }
        }
        for c, emb in zip(batch, embeddings)
    ]
    index.upsert(vectors=vectors)

print(f"Successfully ingested {len(chunks)} documentation chunks!")

Using with LangChain

import json
from langchain_core.documents import Document
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

documents = []
with open("CHUNKS_JSONL.jsonl", "r") as f:
    for line in f:
        item = json.loads(line)
        documents.append(
            Document(
                page_content=item["content"],
                metadata={
                    "url": item["url"],
                    "title": item["title"],
                    "section": item["section"],
                    "chunkId": item["chunkId"]
                }
            )
        )

vector_store = Chroma.from_documents(
    documents=documents,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    persist_directory="./chroma_docs_db"
)

---

🛠️ Local Development & Testing

Installation

git clone https://github.com/your-org/docs2rag.git
cd docs2rag
npm install

Build & Run Locally

# Build TypeScript
npm run build

# Run unit and integration tests
npm test

# Run actor locally using tsx
npm run dev

---

🚢 Deploying to Apify Store

1. Install the Apify CLI:

   $npm install -g apify-cli
   

2. Log in to your Apify account:

   $apify login
   

3. Push and publish the Actor:

   $apify push
   

---

🛡️ Security & Responsible Crawling

• Docs2RAG is designed strictly for crawling public documentation.
• It blocks requests to private IPv4/IPv6 address spaces, localhost, and cloud metadata services (169.254.169.254).
• It complies with robots.txt by default and enforces rate limits and maximum concurrency.
• Ensure you have permission to crawl and index the target documentation website.

---

📄 License

Apache-2.0 License