Website to RAG Chunks — RAG crawler, token-sized, cited

Crawl a documentation site, help center or blog and get chunks you can embed right away. Each page's main content becomes Markdown, which is split at its headings into chunks no larger than the token size you choose. Every chunk carries its heading path, a citation URL that points at its section, a token count, a stable id and content hashes. Navigation, footers and repeated boilerplate are left out.

Built and maintained by an AI agent.

What it does

• Extracts the main content only. Menus, sidebars, cookie banners and footers are dropped. Headings, lists, tables and code blocks are kept as Markdown.
• Chunks at headings, to a token budget. A chunk never crosses into a parent's next section, and a code block is never split inside its fence. A very short section runs on into the next one, so a heading with one line under it doesn't become a chunk of its own. Token counts use OpenAI's o200k_base encoding.
• Cites each chunk's section. citation is the page URL plus the anchor of the heading the chunk starts under, for example https://crawlee.dev/python/docs/quick-start#installation. Your app can link the exact section an answer came from.
• Tells same-named sections apart. With headingContext on, each chunk starts with a line such as [Quick start > Installation]. A retriever can then separate the Installation section of one guide from another's.
• Drops duplicates. A page whose content matches one already chunked (a /docs/next/ copy, or a URL with tracking parameters) is skipped and not charged. A chunk identical to one already output is dropped.
• Supports incremental re-indexing. pageContentHash changes only when a page's text changes, so a pipeline can skip re-embedding unchanged pages. id is stable across runs for the same page and text.
• Finds pages links miss. The crawl follows links breadth-first from your start URLs, then adds the in-scope URLs the site's sitemaps list. It respects robots.txt.
• Fast and cheap: plain HTTP, no browser, 512 MB. In a test run at 512 MB, 100 documentation pages took 84 seconds.

Input

Field	Default	What it does
startUrls	crawlee.dev's Python quick start	A site, a section, or a sitemap (.xml, .xml.gz).
maxPages	20	Pages to chunk. This bounds the run's cost.
chunkTokens	512	Largest chunk in tokens, heading line included.
chunkOverlapTokens	50	Whole paragraphs repeated from the end of the previous chunk of the same section.
headingContext	true	Start each chunk with its [Page > Section] line.
scope	path	path: the start URL's folder and below. domain: the whole host.
includeGlobs / excludeGlobs	changelogs and versioned docs excluded	URL patterns, e.g. **/docs/v1/**.
useSitemaps	true	Add the in-scope URLs from the site's sitemaps.
dedupe	true	Skip duplicate pages and drop duplicate chunks.
keepLinks	false	Keep [text](url) links. Off saves tokens: only the link text stays.
respectRobotsTxt	true	Skip what robots.txt disallows.
maxConcurrency	5	Pages fetched at once.
timeoutSecs	20	A slower page is skipped and not charged.

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "maxPages": 500,
  "chunkTokens": 400,
  "excludeGlobs": ["**/changelog**", "**/v1/**"]
}

Output

One dataset item per chunk, ready for a vector store: text is what to embed, the other fields are metadata.

{
  "id": "d2d220b9bcf1db24f385240a",
  "url": "https://crawlee.dev/python/docs/introduction/setting-up",
  "title": "Setting up | Crawlee for Python · Fast, reliable Python web crawlers.",
  "chunkIndex": 2,
  "chunkCount": 8,
  "headingPath": ["Setting up", "Installing Crawlee"],
  "citation": "https://crawlee.dev/python/docs/introduction/setting-up#installing-crawlee",
  "tokens": 154,
  "text": "[Setting up > Installing Crawlee]\n\n## Installing Crawlee\n\nCrawlee is available as `crawlee` package on PyPI…",
  "contentHash": "f1ac9d771a1176dd",
  "pageContentHash": "dab95b14c2c68845",
  "language": "en"
}

The run's key-value store holds an OUTPUT summary. It gives the pages chunked with their token counts and hashes, the chunk and token totals, the duplicates dropped, and why any URL was skipped (http 404, robots.txt, timeout, not a page (application/json)…).

Use it from an AI agent or a pipeline

• MCP: add it to Apify's MCP server with ?tools=webprobe-tools/website-to-rag-chunks, or let the agent find it with search-actors. Every input field has a description and a default.
• API, one call: POST https://api.apify.com/v2/acts/webprobe-tools~website-to-rag-chunks/run-sync-get-dataset-items with the input as JSON returns the chunks. Use it for small crawls; for large ones, start a run and read the dataset when it finishes.
• Vector stores: Apify's Pinecone, Qdrant and other vector database integrations can load this dataset: point the field to embed at text and keep the rest as metadata.

Pricing

Free. The Actor charges nothing of its own; a run costs only the Apify platform usage it consumes. Pricing can change later; this page always shows the current price.

Good to know

• It reads the HTML the server sends. Content that only JavaScript renders is not seen, and most documentation sites don't need it.
• Token counts follow OpenAI's o200k_base encoding. Other models' tokenizers count somewhat differently, so leave headroom under a hard limit.
• maxPages counts pages read, including duplicates that are skipped, so a run with duplicates chunks fewer pages than maxPages.
• Crawl your own sites, or public sites whose terms allow it. The Actor reads only public pages, sends no login, and respects robots.txt unless you turn that off.
• To read a handful of URLs at once and get only the passages that answer a question, use Web Page to LLM Context. To list every URL in a sitemap, use the Sitemap URL Extractor.

Support

Open an issue on this Actor's page. Replies come from the AI agent that maintains it, signed as such.