Pricing

from $10.00 / 1,000 document processeds

RAG Docs Extractor - Documentation to Chunks

Turn any documentation site into clean, RAG-ready chunks in a single call. Semantic boundaries, preserved structure, per-chunk metadata.

Pricing

from $10.00 / 1,000 document processeds

Rating

0.0

(0)

Developer

C. K.

Actor stats

Bookmarked

Total users

Monthly active users

5 hours ago

Last modified

RAG Docs Extractor

Turn any documentation site into clean, RAG-ready chunks in a single call. Semantic boundaries, preserved structure, per-chunk metadata (source URL, heading path, token count). No post-processing. Pay per document processed.

What it does

Most doc scrapers give you raw HTML or a single wall of text. You then spend hours cleaning, splitting, and fixing broken context before anything is usable in a vector store. This Actor eliminates that step entirely.

Give it a documentation URL. It crawls the site, strips navigation/chrome, converts to clean markdown, and splits each page into semantically meaningful chunks that respect heading boundaries. Every chunk includes the metadata you need for retrieval: source URL, heading path (so you know where in the doc tree it came from), and token count (so you can plan your embedding budget).

The output drops straight into any vector store or RAG pipeline without cleanup.

Output format

Each chunk in the dataset contains:

Field	Type	Description
`content`	string	The chunk text in markdown or plain text
`heading_path`	string	Hierarchical path, e.g. `"Guide > Installation > Requirements"`
`chunk_index`	integer	Position of this chunk within its source document
`token_count`	integer	Token count (cl100k_base encoding)
`source_url`	string	The URL this chunk was extracted from
`document_title`	string	Page title

Input parameters

Parameter	Type	Default	Description
`startUrl`	string	required	Documentation URL to start crawling from
`maxPages`	integer	50	Maximum pages to crawl
`maxChunkTokens`	integer	512	Target max tokens per chunk
`crawlSameDomain`	boolean	true	Stay within the start URL's domain
`pathPrefix`	string	`""`	Only crawl paths starting with this prefix
`outputFormat`	string	`"markdown"`	`"markdown"` or `"plain_text"`

Example usage

Single page extraction

{
    "startUrl": "https://docs.python.org/3/library/asyncio.html",
    "maxPages": 1
}

Full docs site

{
    "startUrl": "https://fastapi.tiangolo.com/",
    "maxPages": 100,
    "pathPrefix": "/tutorial/",
    "maxChunkTokens": 256
}

Pricing

This Actor uses the pay-per-event model. You are charged per document (page) successfully processed and chunked. No charge for pages that are skipped (empty, non-content).

How the chunking works

HTML cleaning — strips navigation, sidebars, footers, cookie banners, and other non-content elements using a curated set of selectors. Falls back to <article>, <main>, or <body>.
Markdown conversion — converts the cleaned HTML to structured markdown, preserving headings, code blocks, tables, lists, and links.
Semantic splitting — splits on heading boundaries first, then paragraph boundaries, then sentence boundaries. Each chunk inherits the heading hierarchy from its position in the document.
Token counting — uses cl100k_base (the encoding used by GPT-4 and most modern embeddings) for accurate token counts.

Responsible use

This Actor respects robots.txt by default (enforced by Crawlee).
It identifies itself with a descriptive User-Agent header so site owners can identify and block it.
Crawlee's built-in autoscaling keeps request rates reasonable and avoids overloading target servers.
You are responsible for ensuring your use complies with the target site's Terms of Service. Only crawl content you have the right to access and process.

Built with

Crawlee for reliable crawling (robots.txt compliant)
BeautifulSoup for HTML parsing
tiktoken for token counting

Docs-to-RAG Crawler

automation-lab/docs-rag-crawler

Crawl documentation sites (ReadTheDocs, GitBook, Docusaurus, Mintlify) into RAG-ready Markdown/JSON chunks with stable chunk IDs, heading breadcrumbs, word counts, and token estimates.

Stas Persiianenko

RAG-Ready Documentation Scraper

alaricus/rag-docs-markdown-scraper

Scrape documentation to framework-optimized Markdown. Features semantic chunking for LLM, vector database, and RAG pipelines. Parse XML sitemaps easily.

Alaricus

Documentation Crawler for RAG

liquid_bark/docs-crawler-for-rag

Specialized crawler for developer documentation sites. Detects frameworks (Docusaurus, GitBook, ReadTheDocs, MkDocs, Sphinx), extracts clean content, and outputs semantically chunked Markdown optimized for RAG pipelines.

Izz

Docs to Markdown + AI Embeddings → Vector DB Crawler

badruddeen/docs-to-markdown-ai-embeddings---vector-db-crawler

Turn any documentation site into clean Markdown, intelligently chunked content with embeddings (Azure/OpenAI), and directly upsert into MongoDB Atlas, Pinecone, Weaviate, Qdrant, or Milvus — ready for RAG, AI assistants, and semantic search in minutes.

Badruddeen Naseem

5.0

(1)

HTML to Markdown — clean conversion, boilerplate stripping

shoebill-dev27/html-to-markdown

Convert scraped HTML into clean Markdown and plain text: headings, nested lists, links, images, code blocks, blockquotes, and tables. Drops scripts, styles, and structural boilerplate (nav/footer/aside) so only content remains. Pure parsing, no LLM cost.

Shinobu Otani

Docs Markdown Rag Ready Crawler

devwithbobby/docs-markdown-rag-ready-crawler

Turn any documentation site or website into clean, structured markdown—ready for RAG, embeddings, and AI agents.

Dev with Bobby

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Hitman studio

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

Dmitry Goncharov

Document Structure Extractor — Markdown to JSON outline

shoebill-dev27/doc-structure-extractor

Turn Markdown documents into structured JSON: nested heading tree with section text, fenced code blocks, links, parsed tables, and size statistics. Pure parsing, no LLM cost.

Shinobu Otani

Docs & Help Center to RAG JSONL

orbiscribe/docs-help-center-rag-snapshot

Paste a docs or help center URL and get clean Markdown, breadcrumbs, page records, and JSONL chunks for RAG.

Orbiscribe Labs

🏛️ Delaware Corporations Search — DE Entity Registry

nexgendata/delaware-corporations-search-scraper

Search the Delaware Division of Corporations entity registry. For each entity name (partial match), returns matching DE entities — file_number + entity_name. KYC, AML compliance, VC sourcing, M&A intelligence, journalism. Stripe Atlas + every DE C-corp incorporation appears here.