Website to RAG Chunks — Vector-DB Ready Markdown

Point it at a website, documentation portal, help center or blog and get LLM-ready chunks for retrieval-augmented generation:

• main content only (Mozilla Readability) converted to clean Markdown, GFM tables and code blocks kept;
• split by headings and packed to your token budget, with overlap inside a section;
• each chunk carries its heading path (Install > Linux), optional context prefix (Page title > Section), stable chunk ID, content hash, token count, and page metadata (title, description, language, dates, URL, crawl depth).

One dataset row = one chunk → embed and upsert into Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, LangChain or LlamaIndex with no extra processing.

How to run it in 3 steps

1. Add your input. In URLs paste documentation or site URLs to split into chunks — e.g. https://crawlee.dev/js/docs/quick-start. The form is already filled with a working example, so you can skip this step for a first try.
2. Press Start. The prefilled run returns 5 pages in about 3 s and costs about $0.015. Set Max results (cost cap) to cap the cost of bigger runs; you only pay for pages actually delivered.
3. Get the data. Open the Output tab and export JSON, CSV or Excel — or run it from the API, Make, Zapier, n8n or an AI agent (see below).

Features

• Retrieval-quality chunking, not naive fixed-size splits: headings define boundaries, code fences are never cut mid-block, tiny fragments are merged, giant tracking links are dropped.
• Change detection: contentHash lets you re-embed only chunks that changed on scheduled re-crawls.
• HTTP only, no browser: plain HTTP crawler with an in-memory frontier — 36 docs pages → 383 chunks in ~5 s.
• Pay per page, not per chunk: $3 per 1,000 pages, however many chunks they produce. Thin pages (below minWordCount) are free.

Use cases

• Chat with your docs: turn a documentation portal or help center into a support chatbot or internal AI assistant.
• RAG knowledge bases: feed product pages, blogs and policies into Pinecone, Qdrant, pgvector or Weaviate.
• Fresh indexes on a schedule: re-crawl weekly and re-embed only chunks whose contentHash changed.
• LLM fine-tuning & evaluation sets: clean, heading-aware Markdown with token counts.

Input

{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "includeUrlPattern": "/js/docs/",
  "maxDepth": 3,
  "maxPagesPerSite": 500,
  "useSitemap": true,
  "maxTokens": 500,
  "overlapTokens": 50,
  "splitOnHeadings": true,
  "prependContext": true
}

Output (one row per chunk)

Real output (default input)

First 2 of 45 records from a run with the prefilled input (3.8 s on a test machine; long values shortened here):

[
  {
    "chunkId": "9ce446e941a8bbb4-0",
    "url": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "headingPath": [],
    "chunkIndex": 0,
    "chunkCount": 7,
    "text": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.\n\nWith this short tutorial you can start scraping with Crawlee in a min…",
    "tokens": 168,
    "contentHash": "e89d78cb6306e956",
    "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
    "language": "en",
    "publishedAt": null,
    "modifiedAt": null,
    "pageWordCount": 755,
    "depth": 0,
    "scrapedAt": "2026-10-15T12:16:30.483Z"
  },
  {
    "chunkId": "9ce446e941a8bbb4-1",
    "url": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "headingPath": [
      "CheerioCrawler"
    ],
    "chunkIndex": 1,
    "chunkCount": 7,
    "text": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast. > CheerioCrawler\n\nThis is a plain HTTP crawler. It parses HTML using t…",
    "tokens": 78,
    "contentHash": "5a737f329555070e",
    "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
    "language": "en",
    "publishedAt": null,
    "modifiedAt": null,
    "pageWordCount": 755,
    "depth": 0,
    "scrapedAt": "2026-10-15T12:16:30.483Z"
  }
]

{
  "chunkId": "9ce446e941a8bbb4-5",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start | Crawlee for JavaScript",
  "headingPath": ["PlaywrightCrawler"],
  "chunkIndex": 5,
  "chunkCount": 8,
  "text": "Quick Start | Crawlee for JavaScript > PlaywrightCrawler\n\n```\nimport { PlaywrightCrawler } from 'crawlee'; …",
  "tokens": 351,
  "contentHash": "4281dd1f1333b889",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two.",
  "language": "en",
  "pageWordCount": 755,
  "depth": 0
}

Pricing

• $3.00 per 1,000 pages ($0.003 per page).
• The prefilled input is billed for 5 pages → $0.015.
• Example: 1,000 pages → $3.00; 10,000 → $30.00. Set Max results (cost cap) to limit any run.

• Thin pages (below minWordCount) are free; you pay per page, not per chunk.
• Your Apify spending limit is respected exactly: the actor stops charging and finishes cleanly when it is reached. Apify platform usage (compute, proxy) is included in the price.

Use with AI agents (MCP), LangChain, RAG & the API

This actor is a ready-made tool for AI agents: an LLM can call it, wait for the run and read clean JSON back.

MCP server (Claude Desktop, Claude Code, Cursor, VS Code, any MCP client) — add Apify's hosted MCP server with this actor enabled as a tool:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=torix33/website-to-rag-chunks",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

LangChain / LangGraph (pip install langchain-apify, set APIFY_API_TOKEN):

from langchain_apify import ApifyActorsTool, ApifyWrapper
from langchain_core.documents import Document

# 1) give the actor to an agent as a tool
tool = ApifyActorsTool("torix33/website-to-rag-chunks")

# 2) or load results as Documents for a RAG index
loader = ApifyWrapper().call_actor(
    actor_id="torix33/website-to-rag-chunks",
    run_input={"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "includeUrlPattern": "/js/docs/", "maxDepth": 3, "maxPagesPerSite": 500, "useSitemap": True, "maxTokens": 500, "overlapTokens": 50, "splitOnHeadings": True, "prependContext": True},
    dataset_mapping_function=lambda item: Document(page_content=item["text"], metadata={"url": item.get("url")}),
)
docs = loader.load()

REST API — one synchronous call, dataset items in the response:

curl -X POST "https://api.apify.com/v2/acts/torix33~website-to-rag-chunks/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "includeUrlPattern": "/js/docs/", "maxDepth": 3, "maxPagesPerSite": 500, "useSitemap": true, "maxTokens": 500, "overlapTokens": 50, "splitOnHeadings": true, "prependContext": true}'

Also works with the Apify Python/JS clients, Make, Zapier, n8n, Google Sheets, webhooks and schedules. Export as JSON, CSV, Excel, XML or RSS.

RAG tip: Rows are already chunks: embed text, upsert with chunkId as the vector ID, and on scheduled re-crawls re-embed only rows whose contentHash changed.

Tips

• Token counts are estimates (≈4 characters per token) — leave ~10% headroom for your embedding model's limit.
• JavaScript-only sites (content rendered in the browser) may yield little text; this actor is HTTP-only for speed and price.

Limitations

• HTTP only: single-page apps that render content in the browser may produce few or no chunks.
• Login-protected pages are not supported; non-HTML files are skipped.
• Chunk sizes use an estimate of ~4 characters per token; your embedding model's tokenizer will count somewhat differently.

FAQ

Which vector databases does it work with? Any: Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or LangChain/LlamaIndex loaders — one row is one chunk.

Do I pay per chunk? No. You pay per page ($0.003), however many chunks it produces.

How do I keep my index fresh? Schedule the actor and compare contentHash per chunkId; re-embed only changed chunks.