Website to RAG Chunks
Pricing
from $0.25 / 1,000 results
Website to RAG Chunks
Crawl any website and turn its pages into clean, chunked, metadata-rich Markdown records ready for RAG pipelines, vector stores, and custom GPTs.
Website to RAG Chunks
Pricing
from $0.25 / 1,000 results
Crawl any website and turn its pages into clean, chunked, metadata-rich Markdown records ready for RAG pipelines, vector stores, and custom GPTs.
Public web pages to crawl and convert into RAG-ready chunks.
[ { "url": "https://docs.apify.com/platform" }]How many link levels to follow from each start URL. 0 = only the start URLs.
Hard cap on total pages crawled per run.
Only follow links that stay on the same domain as the start URL.
Target maximum characters per chunk (~4 chars per token).
Characters of overlap carried between consecutive chunks for context continuity.
Drop or merge chunks smaller than this size.
Fetch and parse sitemap.xml to discover all crawlable URLs before starting. Respects include/exclude globs.
Additional sitemap URLs to fetch (e.g. https://site.com/sitemap-docs.xml). Auto-discovers /sitemap.xml if empty.
[]Download and extract text from PDF links found during crawl (uses pdf-parse). Adds PDF chunks to dataset.
Limit pages extracted per PDF (0 = all).
Add headingPath array to each chunk showing the H1->H2->H3... ancestry for better citation context.
Extract Open Graph, Twitter Card, JSON-LD, and meta tags as additional chunk metadata.