Docs to Answers Pack — Citable, AI-Ready Chunks

Point it at a documentation site. Get back a corpus your AI can answer from and cite — not a wall of text you still have to prepare.

No API key, no login, no anti-bot.

The problem this actually solves

Every web crawler hands you page text. Then you do the same three jobs by hand, every time:

1. Split it somewhere sensible. Slicing every 1,000 characters cuts sentences in half and separates a heading from the thing it explains.
2. Keep the provenance. If a chunk doesn't know where it came from, your AI cannot cite it — and an answer nobody can verify is worth very little.
3. Throw away the repeats. "Was this page helpful?" appears on all 400 pages. It matches every query and answers none.

This does all three, then adds the one that saves real money:

4. A content hash per chunk, so the next run tells you exactly what changed — and you re-embed only that, instead of paying to embed 400 unchanged pages again.

How the chunking works

Headings first, size second. A heading is the author telling you where one idea ends; that is a far better boundary than any character count.

• Every chunk carries a headingPath — Guides > Authentication > Refresh tokens — and a citationUrl that deep-links to that exact heading.
• Code blocks are never split. Half a code sample is worse than none.
• Tiny sections get merged. An index page with twenty one-line sections would otherwise produce twenty 30-token chunks, and a 30-token chunk cannot answer anything. Neighbours are folded together up to targetChunkChars, and mergedHeadings records everything that went in.

Measured on a real 12-page crawl: median chunk went from ~35 tokens to 233, with only one chunk left under 50.

What you get

Field	Description
text	The chunk, cleaned
headingPath	Guides > Auth > Refresh tokens — the breadcrumb
citationUrl	Deep link to that heading, ready to show a user
chunkId	Stable across runs — the key you store alongside your embedding
contentHash	Changes only when the text changes. This is what makes incremental refresh possible
changeType	added · changed · unchanged · removed (incremental mode)
tokensEstimate · charCount	Budget before you embed
duplicateCount	How many pages this exact text appeared on
mergedHeadings	Which sections were folded together
heading · pageTitle · sourceUrl · position	Ordinary provenance

Input

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "pathPrefix": "/docs",
  "maxPages": 200,
  "targetChunkChars": 400,
  "maxChunkChars": 1500
}

pathPrefix is the single most useful setting — it keeps the marketing site and blog out of your corpus.

Scheduled refresh (the reason to keep it running)

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "incrementalMode": true,
  "changedOnly": true
}

Now each run returns only added and changed chunks. Feed those to your embedder and leave the rest alone. QUALITY_REPORT also lists removedChunkIds so you can delete stale vectors.

Sample output

[
  {
    "chunkId": "a3f19c02b8d7e4516ac2",
    "text": "The Apify platform is the best place to run your scrapers and automations in the cloud...",
    "headingPath": "Apify Academy > Beginner courses > Apify platform",
    "heading": "Apify platform",
    "citationUrl": "https://docs.apify.com/academy#apify-platform",
    "mergedHeadings": "Apify platform | API scraping | Anti-scraping protections",
    "tokensEstimate": 123,
    "charCount": 492,
    "contentHash": "7c1e5b90a2f43d68",
    "duplicateCount": 1,
    "changeType": "added"
  }
]

Typical uses

• RAG chatbot over your own docs — the output is ready to embed, with citations built in.
• Support deflection — index your help centre and answer with a link to the exact section.
• Keeping a corpus fresh — schedule it weekly with changedOnly; embed the delta, not the corpus.
• Migrating docs to a vector DB — stable chunkId means re-runs update rows instead of duplicating.
• Auditing your own docs — duplicateCount shows how much of your site is boilerplate.

Pricing

$1.50 / 1,000 chunks ($0.0015 per result), plus a near-zero start fee. A 200-page docs site is typically 1,500–3,000 chunks. With changedOnly, a weekly refresh usually costs a few cents. Never charged beyond maxItems.

⚠️ Read before you rely on it

• JavaScript-rendered docs will come back empty. This fetches server HTML and does not run JS. If a site builds its content client-side, you'll get the "no text extracted" error rather than silent garbage. Most documentation frameworks (Docusaurus, MkDocs, GitBook, Sphinx, Nextra) server-render and work fine.
• chunkId is stable as long as the document structure holds. Renaming a heading or reordering sections changes the id, and that chunk will be reported as removed + added rather than changed. That's honest rather than clever — pretending otherwise would silently corrupt your index.
• tokensEstimate is chars / 4. Good enough to budget with, not exact, and it varies by tokenizer.
• Boilerplate removal is heuristic. It strips nav/header/footer/sidebar and text repeated across pages. On an unusual layout it may take something you wanted — set dropRepeatedBoilerplate: false and compare.

FAQ

• Do I need an API key? No.
• Does it embed the text for me? No, deliberately. You keep control of the model, the cost and where the vectors live. The output is shaped to hand straight to any embedder.
• What chunk size should I use? 1,500 characters (~375 tokens) suits most embedding models. Raise targetChunkChars for denser prose, lower it for reference material.
• Will it crawl my whole website? By default it stays inside the section of the page you gave it — pointing it at docs.example.com/guides/setup crawls /guides/ only, not the whole domain. Start from the site root, or set pathPrefix yourself, to widen or narrow that.
• Can I run it on several docs sites? Yes. Use a different snapshotKey per corpus so their change tracking stays independent.

Related actors

• Sitemap Extractor — enumerate every URL first, then feed them in.
• Broken Link Resurrector — find dead links in the docs you just indexed.
• AI Crawler Audit — check whether AI crawlers are even allowed to read your docs.