Docs-to-RAG Optimizer

What does Docs to RAG Optimizer do?

Docs to RAG Optimizer turns public developer documentation into clean Markdown, RAG-ready chunks, and llms.txt exports—without the sidebar, navbar, and footer noise that generic website scrapers leave behind.

Give it one or more docs URLs (for example Docusaurus docs). It crawls the site, detects the docs platform, extracts the main content, converts it to Markdown, splits it into embedding-friendly chunks, and writes results to Apify datasets and the key-value store.

It does not log into private docs, parse PDF/DOCX, or upload embeddings to your vector database for you. It prepares the data so your RAG pipeline, chatbot, or AI agent can ingest it.

Actor ID: vamsi-krishna/docs-to-rag-optimizer

---

Why use Docs to RAG Optimizer?

Use this Actor when you need documentation-quality data for:

• AI assistants and support chatbots over product docs
• RAG pipelines (LangChain, LlamaIndex, custom apps)
• Vector databases (Pinecone, Supabase Vector, Weaviate, Qdrant, Chroma, OpenAI vector stores)
• Agent discovery via llms.txt / llms-full.txt
• Scheduled recrawls that skip unchanged pages

Built for docs, not generic web pages

Capability	What you get
Platform cleanup	Docusaurus, GitBook, MkDocs/Material, Mintlify, Sphinx/ReadTheDocs (+ Readability fallback)
Chunking	header-aware (default) or fixed-size token windows
Embedding field	Every chunk includes embeddingText
Dedup / change detection	Content hashes, duplicateOf, optional skipUnchangedPages
Quality signals	qualityScore, qualityWarnings, recommendedAction
Exports	Datasets + JSONL + optional CSV/Markdown + llms.txt
Apify platform	Scheduling, API/MCP runs, monitoring, optional proxy

---

What data can it extract?

You can download results as JSON, JSONL, CSV, Markdown, Excel, or other formats from the Apify Console Output / Storage tabs.

Page records (default dataset)

Field	Type	Description
recordType	string	Always "page"
url / canonicalUrl	string	Page URL
title	string	Page title
cleanMarkdown	string	Cleaned documentation Markdown
tokenCount	number	Approximate tokens (OpenAI-style)
contentHash	string	sha256:… hash for dedupe / recrawl
qualityScore	number	0–100 RAG readiness score
recommendedAction	string	use, review, or skip
metadata.docsPlatform	string	Detected platform (e.g. docusaurus)

Chunk records (chunks dataset)

Field	Type	Description
recordType	string	Always "chunk"
chunkId	string	Stable chunk ID
sourceUrl	string	Parent page URL
headingPath	string[]	Section breadcrumb
embeddingText	string	Text to embed in your vector DB
tokenCount	number	Approximate tokens
previousChunkId / nextChunkId	string | null	Adjacent chunks for continuity

Key-value exports

Key	When	Purpose
pages.jsonl / chunks.jsonl	outputFormats includes json	Batch JSONL pipelines
pages.csv / chunks.csv	outputFormats includes csv	Spreadsheets / no-code tools
llms.txt / llms-full.txt	generateLlmsTxt (default on)	LLM / agent discovery
CONTENT_HASHES	always after a successful crawl	Incremental recrawl map (also persisted in named store docs-rag-content-hashes for skipUnchangedPages)
OUTPUT.json	always	Run summary, counts, warnings
page-md_<hash>.md	outputFormats includes markdown	Per-page Markdown files

---

How to convert documentation to RAG data

1. Open Docs to RAG Optimizer in Apify Console (or call it via API / MCP).
2. Set Start URLs to your public docs (e.g. https://docusaurus.io/docs).
3. Set Max pages (start with 5–20 for a smoke test).
4. Optionally set Include patterns so the crawl stays inside /docs.
5. Keep Generate llms.txt on if you want agent-friendly exports.
6. Click Start, then open Output / Storage for pages, chunks, and exports.
7. Use each chunk’s embeddingText field when building your vector index.

Minimal input example

{
  "startUrls": [{ "url": "https://docusaurus.io/docs" }],
  "maxPages": 20,
  "maxDepth": 2,
  "includePatterns": ["^https://docusaurus\\.io/docs"],
  "excludePatterns": ["/blog/"],
  "outputFormats": ["json"],
  "generateLlmsTxt": true,
  "chunkingEnabled": true,
  "chunkStrategy": "header-aware"
}

See the Input tab in Console for every option. Full field notes are below.

---

How much does it cost?

Pricing is pay-per-event on successfully processed pages (page-processed). This Actor participates in Apify Store discounts, so the price drops on higher subscription plans:

Plan tier	Price / page	Per 1,000 pages
FREE	$0.0015	$1.50
BRONZE	$0.00135	$1.35
SILVER	$0.00115	$1.15
GOLD / PLATINUM / DIAMOND	$0.001	$1.00

• You are not charged per chunk.
• Pages skipped as unchanged (skipUnchangedPages) are not charged.
• Platform compute for the run is included in the event model (no separate CU passthrough). Optional Apify Proxy is still billed separately if you enable it.
• Minimum spendable max total charge is $0.005 (useful for small test runs).

Tip: keep maxPages low while tuning patterns, then raise it for production crawls.

---

Input

See the Input tab for the full schema. Important fields:

Field	Notes
startUrls	Required. Public docs URLs.
sitemapUrls	Optional XML sitemaps (indexes are expanded).
maxPages	1–500. Hard cap for safety.
maxDepth	Link depth from seeds.
includePatterns / excludePatterns	JavaScript regex strings (new RegExp(...)).
crawlOnlyDocs	Skip blog/pricing/login-style paths (default true).
outputFormats	json, markdown, and/or csv.
generateLlmsTxt	Write llms.txt + llms-full.txt (default true).
skipUnchangedPages	Skip write/charge when hash matches prior run.
chunkStrategy	header-aware or fixed-size.
chunkSize / chunkOverlap	Token targets for chunking.
proxyConfiguration	Optional; Apify Proxy is paid—leave off unless needed.
maxConcurrency	1–10 (Playwright uses about half).

Regex pattern tip

{
  "includePatterns": ["^https://developer\\.mozilla\\.org/en-US/docs/Web/JavaScript"],
  "excludePatterns": ["/blog/", "/contributors\\.txt$"]
}

---

Output examples

Page (simplified)

{
  "recordType": "page",
  "url": "https://docusaurus.io/docs",
  "title": "Introduction | Docusaurus",
  "tokenCount": 2189,
  "contentHash": "sha256:…",
  "qualityScore": 95,
  "recommendedAction": "use",
  "metadata": { "docsPlatform": "docusaurus", "language": "en" }
}

Chunk (simplified)

{
  "recordType": "chunk",
  "chunkId": "chunk_abc123_000",
  "sourceUrl": "https://docusaurus.io/docs",
  "sectionTitle": "Getting started",
  "headingPath": ["Introduction", "Getting started"],
  "embeddingText": "Install Docusaurus and create your first docs site…",
  "tokenCount": 392
}

---

For AI agents, MCP, and automation

This section is written so agents can decide how to call the Actor and where to read results.

Capability summary

name: docs-to-rag-optimizer
actorId: vamsi-krishna/docs-to-rag-optimizer
task: crawl public docs → clean Markdown + RAG chunks + llms.txt
billingEvent: page-processed (per successfully processed page)
auth: Authorization Bearer token, or Apify agentic payments (x402 / Skyfire / AGI capped token)
inputs.required: startUrls
outputs.datasets: default (pages), chunks (chunks)
outputs.kv: pages.jsonl, chunks.jsonl, llms.txt, llms-full.txt, OUTPUT.json, CONTENT_HASHES
primaryEmbeddingField: embeddingText

Recommended agent workflow

1. Call the Actor with startUrls, a small maxPages, and tight includePatterns.
2. Wait for run status SUCCEEDED (or read OUTPUT.json for status / warnings).
3. Prefer chunk records’ embeddingText for vector upserts.
4. Prefer llms.txt for discovery indexes; llms-full.txt for a single Markdown corpus.
5. For recrawls, set skipUnchangedPages: true so unchanged pages are not recharged.

JavaScript (Apify Client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('vamsi-krishna/docs-to-rag-optimizer').call({
  startUrls: [{ url: 'https://docusaurus.io/docs' }],
  maxPages: 20,
  includePatterns: ['^https://docusaurus\\.io/docs'],
  generateLlmsTxt: true,
  chunkingEnabled: true,
});

const { items: pages } = await client.dataset(run.defaultDatasetId).listItems();
const kv = client.keyValueStore(run.defaultKeyValueStoreId);
const chunksJsonl = await kv.getRecord('chunks.jsonl');
const llmsTxt = await kv.getRecord('llms.txt');

HTTP (API v2)

Use Authorization: Bearer and the /v2/actors/ prefix (not query-string tokens or /v2/acts/).

curl -X POST "https://api.apify.com/v2/actors/vamsi-krishna~docs-to-rag-optimizer/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docusaurus.io/docs"}],"maxPages":5,"generateLlmsTxt":true}'

API reference: docs.apify.com/api/v2. Agents without a full Apify account can use agentic payments (x402, Skyfire, or temporary tokens from agi.apify.com).

---

Tips for better results

• Start with maxPages: 5 and one include pattern, then scale up.
• Prefer Cheerio-friendly static docs; SPA/docs shells may trigger Playwright (slower, more memory).
• Use skipUnchangedPages on schedules to cut cost on quiet doc sites.
• Filter with recommendedAction (use vs review/skip) before embedding.
• Leave Apify Proxy off unless you hit blocks; it is paid.

---

Limits

• Public docs only (no login / SSO).
• No PDF or DOCX extraction in this version.
• Exact hash dedupe only (not fuzzy near-duplicates).
• Caps: maxPages ≤ 500, maxConcurrency ≤ 10.

---

FAQ

Is this a generic website-to-Markdown scraper?

No. It is optimized for developer documentation platforms and RAG chunk quality (heading paths, code/tables, token counts, quality scores).

Which docs platforms are supported?

Docusaurus, GitBook, MkDocs/Material, Mintlify, Sphinx/ReadTheDocs. Other sites fall back to Readability-based extraction.

Where should I read chunks from?

Use the aliased chunks dataset in the run Storage tab, or download chunks.jsonl from the key-value store. Embed the embeddingText field.

Can I run it from Cursor, Claude, or other AI tools?

Yes—via Apify API, Apify MCP, or agentic payments. Use the Actor ID vamsi-krishna/docs-to-rag-optimizer and the capability summary above.

Something failed or looks wrong?

Check OUTPUT.json for warnings and status. Use the Actor Issues tab for feedback. For programmatic access, see the Console API tab.

---

Legal note

This Actor only crawls content you point it at on public documentation sites. Do not use it to collect personal data or content you are not allowed to process. You are responsible for complying with the target site’s terms, robots rules (honored when respectRobotsTxt is enabled), and applicable law (including GDPR where relevant).

---

Keywords

documentation scraper, docs to Markdown, docs to RAG, RAG chunks, llms.txt, LLM documentation crawler, Docusaurus scraper, GitBook scraper, MkDocs scraper, Mintlify scraper, Sphinx, ReadTheDocs, vector database prep, LangChain, LlamaIndex, embeddingText, JSONL export, Apify Actor, AI agent docs ingestion