Website to RAG Chunks: Markdown, Tokens & Incremental Crawl

Turn any website, documentation portal, PDF or Word file into clean, heading-aware chunks with exact token counts, ready to load into Pinecone, Qdrant, Weaviate, Chroma, pgvector or any LLM pipeline.

$0.25 per 1,000 pages. Chunking included. You pay per page, never per chunk.

What you can do with it

• 🤖 Build a RAG chatbot over your docs – crawl docs.yoursite.com, upsert the chunks, done.
• 🔄 Keep a vector index fresh – schedule a daily run in incremental mode: only new or changed pages are returned and charged.
• 📄 Ingest PDFs and DOCX linked from a site (manuals, policies, papers) in the same run.
• 🧠 Give AI agents a tool – works through Apify's MCP server, so Claude, Cursor or any MCP client can call it.
• 📏 Budget your embeddings – every chunk has its exact token count (cl100k_base or o200k_base).

Why this Actor

	This Actor	Typical crawler	Typical "RAG" Actor
Chunking by headings → paragraphs → sentences	✅	❌ do it yourself	✅
Chunks never exceed your limit	✅	–	⚠️ often
Exact tokens (tiktoken)	✅	❌	⚠️ estimated
Price model	per page	per page	per chunk (10× on long pages)
Browser only when a page comes back empty	✅ auto	manual choice	❌
PDF / DOCX / llms.txt	✅	partial	❌
Incremental (ETag, Last-Modified, sitemap lastmod, content hash)	✅	date filter	❌
Sitemap discovery on by default	✅	✅	❌
Removes menus, footers, cookie banners repeated across pages	✅	✅	❌
Duplicate chunks & near-duplicate pages removed	✅	❌	❌

Quick start

1. Paste one or more URLs in Start URLs (a site, a docs folder, a sitemap or a PDF link).
2. Click Start. The defaults work: 512-token chunks, 64-token overlap, sitemap on, robots.txt respected.
3. Export the dataset as JSON/CSV, or send it straight to your vector database with an Apify integration.

Output: one row per chunk

{
  "id": "5f0c1a7e9b2d4c88a1e3f6b0d2c4e8a1",
  "url": "https://docs.example.com/api/auth",
  "title": "Authentication",
  "headingPath": ["API", "Authentication", "Tokens"],
  "chunkIndex": 2,
  "chunkCount": 5,
  "tokens": 498,
  "text": "API > Authentication > Tokens\n\nAccess tokens expire after one hour...",
  "contentHash": "sha256:9d1e…",
  "lastModified": "2026-10-30",
  "language": "en",
  "source": "html",
  "crawledAt": "2026-11-05T10:12:44+00:00"
}

• id is stable per URL and chunk position, so re-runs upsert instead of duplicating.
• source is html, browser, pdf, docx, txt or md.
• Each chunk starts with its section path, which improves retrieval quality.
• Set Output to pages for one row per page with full Markdown, or both.

Every run also saves to its key-value store:

• llms.txt – an index of the crawled site in the llms.txt format.
• content.md – the whole site as a single Markdown file.
• SUMMARY – pages returned, browser pages, documents, tokens, unchanged/skipped/failed URLs and charged events.

Pricing (pay per event)

Event	Price	When
Page	$0.25 / 1,000	HTML, TXT or Markdown page with useful text, chunking included
Browser page	$2.00 / 1,000	Page that needed JavaScript rendering (only when HTTP came back empty)
Document	$1.00 / 1,000	PDF or DOCX file converted
Actor start	$0.05 / 1,000 runs	Apify's standard start fee

Never charged: errors, 404s, empty pages, pages blocked by robots.txt or noindex, duplicates, and pages unchanged since your last incremental run. The run stops cleanly at your Max pages or at the spending limit you set.

Example: a 2,000-page documentation site costs about $0.50. Re-crawling it daily in incremental mode usually costs a few cents.

Incremental mode (keep your index in sync)

Turn on Incremental mode and schedule the Actor (Apify → Schedules). Each run:

1. Re-uses the ETag / Last-Modified headers and sitemap lastmod dates from the previous run to avoid downloading what did not change.
2. Compares a content fingerprint of every downloaded page.
3. Returns and charges only new or changed pages.

State is kept in a named key-value store in your account (derived from your start URLs, or set State store name).

Use with your stack

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("nacenempresas/rag-website-chunks").call(run_input={
    "startUrls": [{"url": "https://docs.example.com"}],
    "chunkSize": 512,
    "incremental": True,
})
for chunk in client.dataset(run["defaultDatasetId"]).iterate_items():
    vector_db.upsert(id=chunk["id"], text=chunk["text"], metadata={"url": chunk["url"], "section": chunk["headingPath"]})

LangChain

from langchain_apify import ApifyDatasetLoader
from langchain_core.documents import Document

loader = ApifyDatasetLoader(
    dataset_id="<DATASET_ID>",
    dataset_mapping_function=lambda c: Document(page_content=c["text"], metadata={"source": c["url"], "section": " > ".join(c["headingPath"])}),
)
docs = loader.load()  # already chunked – no text splitter needed

No code: use Apify's Pinecone, Qdrant or Make/Zapier/n8n integrations on the dataset.

Input options

Field	Default	What it does
Start URLs	–	Sites, folders, sitemaps or document links
Max pages	100	Pages/documents returned and charged per run
Chunk size / overlap	512 / 64 tokens	Hard upper limit per chunk and overlap inside a section
Tokenizer	cl100k_base	o200k_base for GPT-4o and newer, or a fast approximation
Output	chunks	chunks, pages or both
Incremental mode	off	Only new or changed pages
Use sitemap	on	Finds pages that are not linked
Stay within path	on	Only crawl under the start folder
Include / exclude globs	–	e.g. https://site.com/docs/**, **/changelog/**
Linked documents	PDF, DOCX, TXT, MD	Converted and chunked too
JavaScript rendering	auto	Browser only for pages that come back empty
Remove boilerplate	on	Drops repeated menus/footers, duplicate chunks and near-duplicate pages
Respect robots.txt	on	Skips disallowed URLs
Proxy	off	Use your own or Apify Proxy for sites that block datacenter IPs

FAQ

Does it handle JavaScript sites? Yes. Every page is fetched over fast HTTP first; only pages that come back (almost) empty are re-rendered in headless Chromium and billed as browser page. Set JavaScript rendering to always for apps that need it everywhere.

Why per page and not per chunk? Per-chunk pricing punishes long pages and small chunk sizes. Here a long page costs the same as a short one, so you can pick the chunk size that is best for retrieval, not for your bill.

Is the token count exact? Yes, it uses OpenAI's tiktoken with the encoding you choose. Most embedding models tokenize similarly, so it is a reliable budget for any provider.

Can it log in or bypass paywalls? No. It reads public pages only and respects robots.txt by default.

Is it legal to crawl a site? You decide which sites to read and are responsible for how you use the data. Respect each site's terms and copyright. The Actor does not search for or extract personal data.

Support

Found a page that comes back wrong? Open an issue on the Issues tab with the URL – we answer within 48 hours. Changes are listed in the changelog.