# Docs2rag (`thebyteguy/docs2rag`) Actor

Crawl public docs and convert them into AI-ready RAG datasets. Docs2RAG strips navigation noise, preserves code blocks & tables, and exports clean Markdown and structured chunks ready for vector DBs like Pinecone, Qdrant, & Chroma with zero API keys required.

- **URL**: https://apify.com/thebyteguy/docs2rag.md
- **Developed by:** [Chirayu M](https://apify.com/thebyteguy) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Docs2RAG — Documentation-to-RAG Dataset Builder

> **Crawl public documentation websites and export clean Markdown, structured JSON, and RAG-ready chunks for AI agents, LLM applications, and vector databases.**

***

### 💡 Why Docs2RAG?

Documentation websites are essential sources of truth for developers and AI agents. However, raw HTML contains navigation bars, sidebars, cookie banners, footers, search dialogs, and repetitive layout boilerplate that degrades vector retrieval and pollutes LLM context windows.

**Docs2RAG** solves this by:

1. **Intelligently stripping layout noise** (navbars, sidebars, cookie banners, footers) while keeping the core semantic article body.
2. **Preserving critical technical structures**: fenced code blocks with language tags, GitHub Flavored Markdown (GFM) tables, lists, and heading hierarchies.
3. **Producing section-aware RAG chunks**: chunked by heading paths (`["Getting Started", "Installation"]`) without truncating code blocks midway.
4. **Providing deterministic identifiers and content hashes** for incremental sync and deduplication.
5. **Exporting ready-to-ingest formats**: Apify Dataset, Key-Value Store Markdown files (`DOC_*.md`), JSON, and newline-delimited JSON (`JSONL`) ready for Pinecone, Qdrant, Weaviate, Chroma, Supabase, LangChain, and LlamaIndex.

***

### 🚀 Features

- **Semantic Content Extraction**: Automatic detection of `<main>`, `<article>`, `#content`, `.docs-content`, `.markdown-body` with customizable CSS selectors.
- **Noise & Boilerplate Stripping**: Removes banners, headers, footers, sidebars, cookie notices, and ads.
- **High-Fidelity Markdown**: Preserves ATX headings, bold/italics, links, GFM tables, and fenced code blocks with language specifiers.
- **Hierarchy-Aware RAG Chunking**: Chunks preserve section context and breadcrumbs, respect token boundaries, and prevent code block splitting.
- **Sitemap & Deep Crawling**: Discovers pages via `/sitemap.xml` and internal links with configurable depth and pattern filters.
- **Deduplication & Incremental Hashing**: Canonical URL normalization, query parameter cleaning, and SHA-256 content hashing.
- **Zero Mandatory API Keys**: Runs locally or in the cloud without requiring OpenAI or external embedding API keys.
- **Security & SSRF Protection**: Restricts schemes (`http`/`https`), blocks private IP ranges, loopback addresses (`127.0.0.1`, `::1`), and cloud metadata services (`169.254.169.254`).

***

### ⚙️ Actor Input Configuration

| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `startUrls` | `Array` | *Required* | Array of entry documentation URLs (e.g. `["https://docs.docker.com/get-started/"]`). |
| `maxPages` | `Integer` | `100` | Maximum number of documentation pages to crawl. |
| `maxDepth` | `Integer` | `10` | Maximum link crawl depth from start URLs. |
| `includeUrlPatterns` | `Array<String>`| `[]` | Regex or glob patterns to include (e.g. `["/docs/.*"]`). |
| `excludeUrlPatterns` | `Array<String>`| `[]` | Regex or glob patterns to exclude (e.g. `[".*release-notes.*"]`). |
| `sameDomainOnly` | `Boolean` | `true` | Restrict crawl strictly to the same domain as start URL. |
| `respectRobotsTxt` | `Boolean` | `true` | Comply with `robots.txt` directives. |
| `maxConcurrency` | `Integer` | `5` | Maximum concurrent request workers. |
| `requestTimeoutSecs` | `Integer` | `30` | Request timeout per page in seconds. |
| `chunkSize` | `Integer` | `1200` | Target character size per text chunk. |
| `chunkOverlap` | `Integer` | `200` | Character overlap between consecutive chunks. |
| `minContentLength` | `Integer` | `100` | Minimum character length for a page to be processed. |
| `includeCode` | `Boolean` | `true` | Preserve code snippets and syntax highlighting. |
| `includeImages` | `Boolean` | `false` | Preserve image tags with alt text and URLs. |
| `outputFormats` | `Array<String>`| `["MARKDOWN", "JSON", "JSONL"]` | Formats to generate. |
| `deduplicateContent`| `Boolean` | `true` | Skip duplicate pages with matching content hashes. |
| `crawlSitemap` | `Boolean` | `true` | Automatically discover URLs from `/sitemap.xml`. |
| `generatePageSummaries`| `Boolean` | `false` | Generate extractive summaries for each page. |
| `contentSelectors` | `Array<String>`| `[]` | Optional custom CSS selectors for main content. |
| `removeSelectors` | `Array<String>`| `[]` | Optional custom CSS selectors to remove. |

***

### 📋 Example Actor Input

```json
{
  "startUrls": [
    { "url": "https://docs.docker.com/get-started/" }
  ],
  "maxPages": 50,
  "maxDepth": 5,
  "chunkSize": 1000,
  "chunkOverlap": 150,
  "includeCode": true,
  "outputFormats": ["MARKDOWN", "JSON", "JSONL"],
  "crawlSitemap": true,
  "deduplicateContent": true
}
```

***

### 📦 Output Formats & Schemas

#### 1. Default Apify Dataset

Records in the dataset include both full `PageRecord` items and `ChunkRecord` items.

##### Sample `ChunkRecord`

```json
{
  "recordType": "chunk",
  "chunkId": "guide-installation-c0-a1b2c3d4",
  "pageId": "guide-installation-9e8d7c6b5a4f3e2d",
  "url": "https://docs.example.com/guide/installation",
  "title": "Installation Guide — MyDocs",
  "section": "Prerequisites",
  "headingPath": ["Installation Guide", "Prerequisites"],
  "chunkIndex": 0,
  "content": "Before installing the SDK, ensure you have Node.js 18 or higher installed on your system.",
  "contentType": "text",
  "tokenCount": 18,
  "contentHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "metadata": {
    "sourceDomain": "docs.example.com",
    "language": "en",
    "characterCount": 89,
    "wordCount": 15,
    "crawledAt": "2026-10-10T12:00:00.000Z"
  }
}
```

##### Sample `PageRecord`

```json
{
  "recordType": "page",
  "pageId": "guide-installation-9e8d7c6b5a4f3e2d",
  "url": "https://docs.example.com/guide/installation",
  "canonicalUrl": "https://docs.example.com/guide/installation",
  "title": "Installation Guide — MyDocs",
  "description": "Step by step instructions to install and configure MyDocs.",
  "sourceDomain": "docs.example.com",
  "language": "en",
  "headings": [
    { "level": 1, "text": "Installation Guide" },
    { "level": 2, "text": "Prerequisites" },
    { "level": 2, "text": "Quick Start" }
  ],
  "contentHash": "8f481c47be3c4dd2d4...",
  "crawledAt": "2026-10-10T12:00:00.000Z",
  "httpStatus": 200,
  "wordCount": 420,
  "characterCount": 2650,
  "chunkCount": 3,
  "content": "Full extracted plain text...",
  "contentMarkdown": "# Installation Guide\n\n## Prerequisites\n\nBefore installing...",
  "summary": "This document covers Installation Guide. Key sections include: Prerequisites, Quick Start."
}
```

#### 2. Key-Value Store Artifacts

- `DOC_{pageId}.md`: Individual Markdown files with frontmatter.
- `CHUNKS_JSONL`: Newline-delimited JSON file of all chunks for direct vector ingestion.
- `PAGES_JSONL`: Newline-delimited JSON file of all pages.
- `MANIFEST`: Comprehensive run statistics, timings, and configuration snapshot.

***

### 🔌 Vector Database Integration Examples

#### Ingesting `CHUNKS_JSONL` into Vector Databases

```python
import json
from openai import OpenAI
from pinecone import Pinecone

## 1. Initialize Clients
openai_client = OpenAI(api_key="your-openai-api-key")
pc = Pinecone(api_key="your-pinecone-api-key")
index = pc.Index("docs-knowledge-base")

## 2. Read Chunks JSONL exported from Docs2RAG
with open("CHUNKS_JSONL.jsonl", "r") as f:
    chunks = [json.loads(line) for line in f]

## 3. Generate Embeddings & Upsert in Batches
batch_size = 100
for i in range(0, len(chunks), batch_size):
    batch = chunks[i:i + batch_size]
    texts = [c["content"] for c in batch]
    
    # Generate embeddings with text-embedding-3-small
    res = openai_client.embeddings.create(input=texts, model="text-embedding-3-small")
    embeddings = [r.embedding for r in res.data]
    
    # Prepare Pinecone vectors
    vectors = [
        {
            "id": c["chunkId"],
            "values": emb,
            "metadata": {
                "url": c["url"],
                "title": c["title"],
                "section": c["section"],
                "headingPath": " > ".join(c["headingPath"]),
                "content": c["content"],
                "sourceDomain": c["metadata"]["sourceDomain"]
            }
        }
        for c, emb in zip(batch, embeddings)
    ]
    index.upsert(vectors=vectors)

print(f"Successfully ingested {len(chunks)} documentation chunks!")
```

#### Using with LangChain

```python
import json
from langchain_core.documents import Document
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

documents = []
with open("CHUNKS_JSONL.jsonl", "r") as f:
    for line in f:
        item = json.loads(line)
        documents.append(
            Document(
                page_content=item["content"],
                metadata={
                    "url": item["url"],
                    "title": item["title"],
                    "section": item["section"],
                    "chunkId": item["chunkId"]
                }
            )
        )

vector_store = Chroma.from_documents(
    documents=documents,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    persist_directory="./chroma_docs_db"
)
```

***

### 🛠️ Local Development & Testing

#### Installation

```bash
git clone https://github.com/your-org/docs2rag.git
cd docs2rag
npm install
```

#### Build & Run Locally

```bash
## Build TypeScript
npm run build

## Run unit and integration tests
npm test

## Run actor locally using tsx
npm run dev
```

***

### 🚢 Deploying to Apify Store

1. Install the Apify CLI:
   ```bash
   npm install -g apify-cli
   ```
2. Log in to your Apify account:
   ```bash
   apify login
   ```
3. Push and publish the Actor:
   ```bash
   apify push
   ```

***

### 🛡️ Security & Responsible Crawling

- Docs2RAG is designed strictly for crawling **public documentation**.
- It blocks requests to private IPv4/IPv6 address spaces, localhost, and cloud metadata services (`169.254.169.254`).
- It complies with `robots.txt` by default and enforces rate limits and maximum concurrency.
- Ensure you have permission to crawl and index the target documentation website.

***

### 📄 License

Apache-2.0 License

# Actor input Schema

## `startUrls` (type: `array`):

One or more documentation entry point URLs to begin crawling.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl and extract. Default is 100.

## `maxDepth` (type: `integer`):

Maximum link crawl depth from the start URLs. Default is 10.

## `includeUrlPatterns` (type: `array`):

Optional list of regex or glob patterns. Only URLs matching at least one pattern will be crawled.

## `excludeUrlPatterns` (type: `array`):

Optional list of regex or glob patterns. Matching URLs will be skipped.

## `sameDomainOnly` (type: `boolean`):

If true, only links on the same host domain as the start URL will be crawled.

## `respectRobotsTxt` (type: `boolean`):

Whether to fetch and respect the target site's robots.txt rules.

## `maxConcurrency` (type: `integer`):

Maximum number of concurrent requests. Keep safe to avoid overloading target servers.

## `requestTimeoutSecs` (type: `integer`):

Maximum timeout in seconds for each HTTP request.

## `removeSelectors` (type: `array`):

Custom CSS selectors for elements to remove from the page prior to extraction (e.g., banner, cookie popups).

## `contentSelectors` (type: `array`):

Custom CSS selectors to locate the main documentation body. If provided, the first match is extracted.

## `chunkSize` (type: `integer`):

Target size for each text chunk (in characters, or tokens if token-aware chunking). Default is 1200.

## `chunkOverlap` (type: `integer`):

Overlap between consecutive chunks (in characters, or tokens). Default is 200.

## `minContentLength` (type: `integer`):

Minimum character count for extracted page content to be kept. Prevents saving empty or stub pages.

## `includeCode` (type: `boolean`):

Whether to preserve code blocks and syntax-highlighted snippets in Markdown and chunks.

## `includeImages` (type: `boolean`):

Whether to preserve image markdown syntax with alt text and URLs.

## `outputFormats` (type: `array`):

Supported output formats to produce: MARKDOWN, JSON, JSONL.

## `deduplicateContent` (type: `boolean`):

Avoid processing duplicate pages with identical content hashes.

## `crawlSitemap` (type: `boolean`):

Automatically discover documentation URLs via /sitemap.xml if available.

## `generatePageSummaries` (type: `boolean`):

Optional extractive page summary generation (does not require external API keys).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.docker.com/get-started/"
    }
  ],
  "maxPages": 100,
  "maxDepth": 10,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30,
  "removeSelectors": [],
  "contentSelectors": [],
  "chunkSize": 1200,
  "chunkOverlap": 200,
  "minContentLength": 100,
  "includeCode": true,
  "includeImages": false,
  "outputFormats": [
    "MARKDOWN",
    "JSON",
    "JSONL"
  ],
  "deduplicateContent": true,
  "crawlSitemap": true,
  "generatePageSummaries": false
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.docker.com/get-started/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("thebyteguy/docs2rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.docker.com/get-started/" }] }

# Run the Actor and wait for it to finish
run = client.actor("thebyteguy/docs2rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.docker.com/get-started/"
    }
  ]
}' |
apify call thebyteguy/docs2rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thebyteguy/docs2rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/hcdPVxsl5iG83flKa/builds/Cv3w5hAildRY5Q8rP/openapi.json