Website to RAG Chunks

Crawl any website and get back clean, chunked, metadata-rich Markdown ready to embed — sized chunks with overlap, heading ancestry for citations, and token estimates. Built for RAG pipelines, vector stores and custom GPTs.

What it does

Getting a website into a vector database normally means writing a crawler, stripping boilerplate, converting to Markdown, chunking sensibly and preserving enough metadata to cite sources. This Actor does all of it and hands you records that go straight into Pinecone, Qdrant, Weaviate, pgvector, Chroma or a custom GPT's knowledge file.

Crawl scope is yours to control: depth, page cap, same-domain restriction, include/exclude globs, and optional sitemap discovery so you can enumerate a documentation site properly instead of hoping links cover it. PDFs linked from the crawl can be extracted too.

Features

• Chunking that respects structure — chunkSize with chunkOverlap, and minChunkChars to merge away useless fragments.
• Heading path per chunk — headingPath records the H1→H2→H3 ancestry, so citations can say exactly where text came from.
• Token estimates — tokenEstimate on every chunk for budgeting embedding and context costs.
• Sitemap crawling — useSitemap enumerates a site properly, honouring your include/exclude globs.
• PDF extraction — extractPdfs pulls text out of linked PDFs, with a per-file page cap.
• Glob scoping — includeGlobs / excludeGlobs to crawl /docs/** and skip /blog/**.
• Metadata enrichment — Open Graph, Twitter Card, JSON-LD and meta tags attached to chunks.
• Content hashing — contentHash per chunk makes deduplication and incremental re-indexing trivial.

What people use it for

• Build a RAG knowledge base from product or API documentation.
• Feed a custom GPT or assistant with your own site's content.
• Populate a vector store for semantic search across a docs portal.
• Create an internal Q&A bot over a knowledge base or handbook.
• Incremental re-indexing — hash comparison shows exactly which chunks changed.

Choosing chunk size and overlap

There is no universally right answer, but these are sound starting points:

Use case	chunkSize	chunkOverlap
Precise Q&A over docs	500–800	100
General RAG (default)	1000	150
Long-form summarisation	2000+	200

Roughly four characters make one token, so chunkSize: 1000 lands near 250 tokens. Overlap exists so a sentence split across a boundary is still retrievable from at least one chunk — without it, answers spanning a boundary get lost.

Crawling a documentation site properly

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "useSitemap": true,
  "includeGlobs": ["https://docs.example.com/**"],
  "excludeGlobs": ["**/changelog/**", "**/blog/**"],
  "maxPages": 500,
  "chunkSize": 1000,
  "chunkOverlap": 150
}

useSitemap is the difference between crawling a docs site and hoping your link graph reached every page.

Keeping an index fresh

contentHash identifies chunk content exactly. Store it alongside your vectors, re-run the Actor on a schedule, and re-embed only chunks whose hash changed — which is usually a tiny fraction and keeps embedding costs near zero.

Input

startUrls is required. Defaults produce a small, safe crawl — raise maxCrawlDepth and maxPages deliberately once you know the shape of the site.

Field	Type	Default	What it does
startUrls (required)	array	[{"url": "https://docs.apify.com/platform"}]	Public web pages to crawl and convert into RAG-ready chunks.
maxCrawlDepth	integer	1	How many link levels to follow from each start URL. 0 = only the start URLs. Range 0–10.
maxPages	integer	50	Hard cap on total pages crawled per run. Range 1–10000.
sameDomainOnly	boolean	true	Only follow links that stay on the same domain as the start URL.
includeGlobs	array	[]	Only enqueue URLs matching these glob patterns (e.g. https://site.com/docs/**).
excludeGlobs	array	[]	Skip URLs matching these glob patterns.
chunkSize	integer	1000	Target maximum characters per chunk (~4 chars per token). Range 200–20000.
chunkOverlap	integer	150	Characters of overlap carried between consecutive chunks for context continuity. Range 0–5000.
minChunkChars	integer	200	Drop or merge chunks smaller than this size. Range 1–5000.
useSitemap	boolean	false	Fetch and parse sitemap.xml to discover all crawlable URLs before starting. Respects include/exclude globs.
sitemapUrls	array	[]	Additional sitemap URLs to fetch (e.g. https://site.com/sitemap-docs.xml). Auto-discovers /sitemap.xml if empty.
extractPdfs	boolean	false	Download and extract text from PDF links found during crawl (uses pdf-parse). Adds PDF chunks to dataset.
maxPdfPages	integer	50	Limit pages extracted per PDF (0 = all). Range 0–500.
includeHeadingPath	boolean	true	Add headingPath array to each chunk showing the H1->H2->H3... ancestry for better citation context.
enrichMetadata	boolean	true	Extract Open Graph, Twitter Card, JSON-LD, and meta tags as additional chunk metadata.
proxyConfiguration	object	see below	Apify Proxy configuration for blocked sites.

Input example

{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 0,
  "maxPages": 1,
  "sameDomainOnly": true,
  "chunkSize": 500,
  "chunkOverlap": 50,
  "minChunkChars": 50,
  "includeHeadingPath": true,
  "enrichMetadata": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "proxyUrls": []
  },
  "useSitemap": false,
  "extractPdfs": false,
  "maxPdfPages": 50
}

Output

One record per chunk, carrying its source URL and page title, its position (chunkIndex of totalChunks), the Markdown text, character count, token estimate, content hash and crawl timestamp.

Every dataset record contains: url, title, chunkIndex, totalChunks, text, charCount, tokenEstimate, contentHash, crawledAt, headingPath, sourceType, metadata.

Output example

A real record from a run of this Actor:

{
  "url": "https://example.com/",
  "title": "Example Domain",
  "chunkIndex": 0,
  "totalChunks": 1,
  "text": "This domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "charCount": 149,
  "tokenEstimate": 38,
  "contentHash": "a80242055d7489b4ce56b3193e7521cd92ef34fdc5b6aa666d7bef8cc29ccfd5",
  "crawledAt": "2026-08-22T15:28:17.922Z",
  "headingPath": [
    "Example Domain"
  ],
  "sourceType": "html",
  "metadata": {
    "viewport": "width=device-width, initial-scale=1"
  }
}

Export the dataset as JSON, CSV, Excel, XML or JSONL from the Console, or pull it programmatically through the Apify API and any of the official clients.

How to use it

1. Click Try for free (or Start if you already have an Apify account).
2. Fill in the input fields described above — the defaults already produce a working run.
3. Press Start and watch the log; results stream into the dataset as they are found.
4. When the run finishes, open the Output/Storage tab and export as JSON, CSV or Excel.

Runs can be scheduled (hourly, daily, weekly) and wired into Slack, Google Sheets, Zapier, Make, webhooks or your own backend through Apify integrations. Everything the Console does is also available over the Apify API.

Proxy configuration

This Actor accepts a standard Apify proxy configuration object. Residential proxy is the default because the target site rate-limits datacenter IP ranges; you can select a specific exit country or supply your own proxy URLs.

{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}

Pricing

This Actor is billed on Apify's pay-per-event model: a small charge when a run starts, plus a charge for each result written to the dataset. You only pay for records you actually receive — a run that finds nothing costs only the start event. Current rates are always shown on the Pricing tab of this page, and the run log prints your usage as it goes.

Free-plan credits from Apify cover a large amount of light usage, so you can evaluate the Actor before committing to anything.

FAQ

What format is the chunk text in?

Markdown. Headings, lists, links and code blocks survive, which both embeds better than raw HTML and reads better when an LLM cites it.

How do I use headingPath for citations?

It's the heading ancestry of the chunk, e.g. ["API Reference", "Authentication", "API keys"]. Store it as metadata and your assistant can cite "API Reference › Authentication › API keys" instead of just a bare URL.

Does it handle JavaScript-rendered sites?

It crawls and extracts page content, and works well on server-rendered and statically generated sites — which covers nearly all documentation. Heavily client-rendered apps with no server HTML may yield thin text; check a small run before committing to a large crawl.

Can I crawl a site that blocks datacenter IPs?

Yes — configure proxyConfiguration with Apify Proxy, RESIDENTIAL group if needed.

Will it crawl the entire internet by accident?

No. sameDomainOnly is on by default, maxCrawlDepth defaults to 1 and maxPages defaults to 50. You have to deliberately widen the scope.

How do PDFs get chunked?

Enable extractPdfs and linked PDFs are downloaded, text-extracted and chunked with the same settings as HTML pages. maxPdfPages caps very long documents.

Other Actors by cynix_dev

Actor	What it does
Dataset Drift & QA Monitor	Stop finding out your scrapers broke three days late. Point this actor at any Apify dataset or JSON endpoint and it watches the …
Page Change Monitor	Monitor web pages for content changes. Diffs each run against the previous snapshot and emits structured change records with …
OpenStreetMap Geocoder	Forward and reverse geocoding via the free Komoot Photon / OpenStreetMap service. No API key, no scraping, ODbL data.
arXiv Papers Extractor	Search arXiv and extract papers as clean typed records: title, abstract, authors, categories, DOI, and direct PDF links.
Page Change Monitor	Monitor web pages for content changes. Diffs each run against the previous snapshot and emits structured change records with …

Legal and responsible use

This Actor collects only publicly available information. You are responsible for how you use the data, including compliance with the target site's Terms of Service, robots directives, copyright, and data protection law such as GDPR and CCPA. Do not use it to gather personal data without a lawful basis.

Support and feedback

Found a bug, hit a site change, or need an extra field? Open a ticket on the Issues tab of this Actor — issues are read and fixed. Feature requests and custom-scraper enquiries are welcome through the same channel.