Website to LLM-ready Markdown – RAG Chunks & Token Counts

Turn any public website, docs site or sitemap into clean Markdown for LLMs, already split into heading-aware RAG chunks with token counts. Navigation, footers, cookie banners and other boilerplate are removed. Load the output straight into Pinecone, Qdrant, Weaviate, pgvector, Chroma, LangChain or LlamaIndex.

$1.50 per 1,000 pages. You pay only for pages that were converted. Failed, empty or blocked pages are listed with a status and are free.

Why this crawler

• ✅ Chunked and token-counted out of the box: every page comes with a chunks array (text, tokenCount, headingPath, chunkId), so you don't need a separate splitting step
• ✅ Real tokenizer counts: o200k_base (GPT-4o, GPT-5) or cl100k_base (GPT-4, text-embedding-3), not a rough guess
• ✅ Fast and cheap: plain HTTP, no headless browser. 3 pages finish in about 5 seconds, compute costs are tiny
• ✅ Smart chunking: chunks follow headings and paragraphs, code blocks stay intact, optional token overlap
• ✅ Polite: respects robots.txt, limited concurrency, own user agent
• ✅ Sitemap support: pass sitemap.xml (also sitemap indexes and .xml.gz) to convert a whole site without crawling

Who it's for: developers building RAG chatbots and AI agents, teams that want their docs in a vector database, and anyone feeding web pages to ChatGPT, Claude or Gemini.

When to use something else: pages that only render content with JavaScript (single-page apps) need a browser-based crawler. This actor is built for the large majority of docs sites, blogs, help centers and marketing sites that serve HTML.

Input

Field	Default	Description
startUrls	https://docs.apify.com/	Start pages, or a sitemap URL
maxPages	3	Maximum pages converted per run
maxDepth	3	Link hops to follow from the start URLs (0 = start URLs / sitemap pages only)
sameDomain	true	Only follow links on the start URL's host
includeUrlPatterns / excludeUrlPatterns	–	Globs like https://docs.example.com/guides/** or plain substrings like /blog/
chunkSize	500	Maximum tokens per chunk
chunkOverlap	0	Tokens repeated from the previous chunk
tokenizer	o200k_base	o200k_base or cl100k_base
includeLinks, includeTables, includeChunks	false, true, true	Output options
respectRobotsTxt	true	Skip URLs disallowed by robots.txt
proxyConfiguration	off	Optional proxy for sites that rate-limit

{
  "startUrls": [{ "url": "https://docs.apify.com/" }],
  "maxPages": 50,
  "includeUrlPatterns": ["https://docs.apify.com/platform/**"],
  "chunkSize": 400,
  "chunkOverlap": 40
}

Output

One dataset item per page. Real example from a test run (Markdown shortened):

{
  "url": "https://docs.apify.com/actors",
  "depth": 1,
  "status": "ok",
  "httpStatus": 200,
  "title": "Actors | Platform | Apify Documentation",
  "description": "Learn how to develop, run and share serverless cloud programs. Create your own web scraping and automation tools and publish them on the Apify platform.",
  "language": "en",
  "canonicalUrl": "https://docs.apify.com/actors",
  "wordCount": 229,
  "tokenCountEstimate": 294,
  "tokenizer": "o200k_base",
  "chunkCount": 1,
  "markdown": "# Actors\n\nActors are serverless cloud programs that take a structured JSON input, perform a task (web scraping, browser automation, data processing, and more), and optionally produce a structured output. ...\n\n## Actor components\n\n- *Dockerfile* which specifies where the Actor's source code is ...",
  "chunks": [
    {
      "chunkIndex": 0,
      "chunkId": "https://docs.apify.com/actors#chunk-0",
      "headingPath": "Actors",
      "headings": ["Actors"],
      "text": "# Actors\n\nActors are serverless cloud programs ...",
      "tokenCount": 294,
      "charCount": 1393
    }
  ]
}

Field	Description
url, loadedUrl, canonicalUrl, depth	Requested URL, URL after redirects, canonical link, crawl depth
status	ok, failed, no_content or blocked_by_robots_txt (only ok is charged)
title, description, language, author, publishedDate, siteName	Page metadata
markdown	Main content as Markdown, boilerplate removed
wordCount, charCount, tokenCountEstimate	Size of the Markdown
chunks[]	chunkIndex, chunkId, headingPath (e.g. Guides > Install > Linux), headings, text, tokenCount, charCount

The dataset has two views: Overview (one row per page) and RAG chunks (one row per chunk).

Pricing

Pay per event: $0.0015 per page converted ($1.50 per 1,000 pages). Pages that fail, are empty or are blocked by robots.txt cost nothing. Because no browser is used, platform compute usage stays very low.

Use it via API, n8n, Make or MCP

API (sync, returns items):

curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~website-to-llm-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.apify.com/"}],"maxPages":10,"chunkSize":500}'

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("fast_data_asap/website-to-llm-markdown").call(run_input={
    "startUrls": [{"url": "https://docs.apify.com/"}], "maxPages": 10,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page.get("chunks", []):
        print(chunk["chunkId"], chunk["tokenCount"], chunk["headingPath"])

n8n / Make / Zapier: use the Apify integration, choose the actor fast_data_asap/website-to-llm-markdown and map the chunks array into your vector store node.

MCP (Claude, Cursor, other AI agents): add https://mcp.apify.com?tools=fast_data_asap/website-to-llm-markdown as an MCP server so your agent can read whole websites as Markdown on demand.

FAQ

How are tokens counted? With the tiktoken encoding you choose. For Claude, Gemini and Llama the counts are a close estimate.

Does it follow robots.txt? Yes, by default. You can switch this off for your own sites.

Can I convert only part of a site? Yes: use includeUrlPatterns, excludeUrlPatterns and maxDepth, or pass a sitemap.

Is it legal? The actor only reads publicly available pages without logging in. Make sure your use of the content respects the site's terms and copyright.