# Docs to Answers Pack — Citable AI-Ready Chunks (`alaudinburki/docs-to-answers`) Actor

Turn a documentation site into a corpus an AI can answer FROM and CITE. Splits on headings rather than character counts, attaches a heading path and deep link to every chunk, drops boilerplate that repeats across pages, and hashes each chunk so you re-embed only what changed.

- **URL**: https://apify.com/alaudinburki/docs-to-answers.md
- **Developed by:** [alaudin burki](https://apify.com/alaudinburki) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Docs to Answers Pack — Citable, AI-Ready Chunks

Point it at a documentation site. Get back a corpus your AI can **answer from and cite** — not a wall
of text you still have to prepare.

**No API key, no login, no anti-bot.**

### The problem this actually solves

Every web crawler hands you page text. Then you do the same three jobs by hand, every time:

1. **Split it somewhere sensible.** Slicing every 1,000 characters cuts sentences in half and separates a
   heading from the thing it explains.
2. **Keep the provenance.** If a chunk doesn't know where it came from, your AI cannot cite it — and an
   answer nobody can verify is worth very little.
3. **Throw away the repeats.** "Was this page helpful?" appears on all 400 pages. It matches every query
   and answers none.

This does all three, then adds the one that saves real money:

4. **A content hash per chunk**, so the next run tells you exactly what changed — and you re-embed only
   that, instead of paying to embed 400 unchanged pages again.

### How the chunking works

**Headings first, size second.** A heading is the author telling you where one idea ends; that is a far
better boundary than any character count.

- Every chunk carries a **`headingPath`** — `Guides > Authentication > Refresh tokens` — and a
  **`citationUrl`** that deep-links to that exact heading.
- **Code blocks are never split.** Half a code sample is worse than none.
- **Tiny sections get merged.** An index page with twenty one-line sections would otherwise produce
  twenty 30-token chunks, and a 30-token chunk cannot answer anything. Neighbours are folded together
  up to `targetChunkChars`, and `mergedHeadings` records everything that went in.

Measured on a real 12-page crawl: median chunk went from **~35 tokens to 233**, with only one chunk
left under 50.

### What you get

| Field | Description |
|---|---|
| **`text`** | The chunk, cleaned |
| **`headingPath`** | `Guides > Auth > Refresh tokens` — the breadcrumb |
| **`citationUrl`** | Deep link to that heading, ready to show a user |
| **`chunkId`** | **Stable across runs** — the key you store alongside your embedding |
| **`contentHash`** | Changes only when the text changes. This is what makes incremental refresh possible |
| **`changeType`** | `added` · `changed` · `unchanged` · `removed` (incremental mode) |
| `tokensEstimate` · `charCount` | Budget before you embed |
| `duplicateCount` | How many pages this exact text appeared on |
| `mergedHeadings` | Which sections were folded together |
| `heading` · `pageTitle` · `sourceUrl` · `position` | Ordinary provenance |

### Input

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "pathPrefix": "/docs",
  "maxPages": 200,
  "targetChunkChars": 400,
  "maxChunkChars": 1500
}
```

`pathPrefix` is the single most useful setting — it keeps the marketing site and blog out of your corpus.

#### Scheduled refresh (the reason to keep it running)

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "incrementalMode": true,
  "changedOnly": true
}
```

Now each run returns **only added and changed chunks**. Feed those to your embedder and leave the rest
alone. `QUALITY_REPORT` also lists `removedChunkIds` so you can delete stale vectors.

### Sample output

```json
[
  {
    "chunkId": "a3f19c02b8d7e4516ac2",
    "text": "The Apify platform is the best place to run your scrapers and automations in the cloud...",
    "headingPath": "Apify Academy > Beginner courses > Apify platform",
    "heading": "Apify platform",
    "citationUrl": "https://docs.apify.com/academy#apify-platform",
    "mergedHeadings": "Apify platform | API scraping | Anti-scraping protections",
    "tokensEstimate": 123,
    "charCount": 492,
    "contentHash": "7c1e5b90a2f43d68",
    "duplicateCount": 1,
    "changeType": "added"
  }
]
```

### Typical uses

- **RAG chatbot over your own docs** — the output is ready to embed, with citations built in.
- **Support deflection** — index your help centre and answer with a link to the exact section.
- **Keeping a corpus fresh** — schedule it weekly with `changedOnly`; embed the delta, not the corpus.
- **Migrating docs to a vector DB** — stable `chunkId` means re-runs update rows instead of duplicating.
- **Auditing your own docs** — `duplicateCount` shows how much of your site is boilerplate.

### Pricing

**$1.50 / 1,000 chunks** (`$0.0015` per result), plus a near-zero start fee. A 200-page docs site is
typically 1,500–3,000 chunks. With `changedOnly`, a weekly refresh usually costs a few cents. Never
charged beyond `maxItems`.

### ⚠️ Read before you rely on it

- **JavaScript-rendered docs will come back empty.** This fetches server HTML and does not run JS. If a
  site builds its content client-side, you'll get the "no text extracted" error rather than silent
  garbage. Most documentation frameworks (Docusaurus, MkDocs, GitBook, Sphinx, Nextra) server-render
  and work fine.
- **`chunkId` is stable *as long as the document structure holds*.** Renaming a heading or reordering
  sections changes the id, and that chunk will be reported as `removed` + `added` rather than
  `changed`. That's honest rather than clever — pretending otherwise would silently corrupt your index.
- **`tokensEstimate` is `chars / 4`.** Good enough to budget with, not exact, and it varies by tokenizer.
- **Boilerplate removal is heuristic.** It strips nav/header/footer/sidebar and text repeated across
  pages. On an unusual layout it may take something you wanted — set `dropRepeatedBoilerplate: false`
  and compare.

### FAQ

- **Do I need an API key?** No.
- **Does it embed the text for me?** No, deliberately. You keep control of the model, the cost and where
  the vectors live. The output is shaped to hand straight to any embedder.
- **What chunk size should I use?** 1,500 characters (~375 tokens) suits most embedding models. Raise
  `targetChunkChars` for denser prose, lower it for reference material.
- **Will it crawl my whole website?** By default it stays inside the section of the page you gave it
  — pointing it at `docs.example.com/guides/setup` crawls `/guides/` only, not the whole domain. Start
  from the site root, or set `pathPrefix` yourself, to widen or narrow that.
- **Can I run it on several docs sites?** Yes. Use a different `snapshotKey` per corpus so their change
  tracking stays independent.

### Related actors

- **Sitemap Extractor** — enumerate every URL first, then feed them in.
- **Broken Link Resurrector** — find dead links in the docs you just indexed.
- **AI Crawler Audit** — check whether AI crawlers are even allowed to read your docs.

# Actor input Schema

## `startUrls` (type: `array`):

Where to start crawling. Point it at a docs homepage or section index.

## `pathPrefix` (type: `string`):

Only crawl URLs whose path starts with this, e.g. /docs. The single most effective way to keep a blog or marketing site out of your corpus.

## `maxPages` (type: `integer`):

Upper bound on pages fetched.

## `followLinks` (type: `boolean`):

Crawl onward from the pages given. Turn off to process only the exact URLs supplied.

## `sameOriginOnly` (type: `boolean`):

Only follow links on the same domain as the start URL. Turn off only if your docs span subdomains.

## `maxChunkChars` (type: `integer`):

Chunks split on headings first and size second, so this is a ceiling rather than a target. ~1500 chars suits most embedding models.

## `minChunkChars` (type: `integer`):

Anything smaller is folded into the previous chunk instead of being emitted as an unanswerable fragment.

## `overlapChars` (type: `integer`):

Repeat this many characters from the previous chunk so a sentence spanning a boundary stays retrievable from both sides.

## `dropRepeatedBoilerplate` (type: `boolean`):

Remove chunks that repeat across pages — "Was this page helpful?", licence footers, the same admonition on 90 pages. These match every query and answer none.

## `nearDuplicateDetection` (type: `boolean`):

Catch boilerplate that differs by a word or two, not just exact repeats.

## `incrementalMode` (type: `boolean`):

Store a hash per chunk and label each one added / changed / unchanged / removed on the next run. Embedding is the expensive step in a RAG pipeline — this lets you re-embed only what actually changed.

## `changedOnly` (type: `boolean`):

With incremental mode on, return just the added and changed chunks. Turns this into a scheduled corpus-refresh job.

## `snapshotKey` (type: `string`):

Key-value store key holding the previous run's hashes. Use different keys to track several corpora independently.

## `concurrency` (type: `integer`):

How many pages to fetch at once. Lower it if the docs site is rate-limiting you.

## `maxItems` (type: `integer`):

Hard cap on results. You are never charged beyond this.

## `proxyConfiguration` (type: `object`):

Optional. Most documentation sites need no proxy.

## `targetChunkChars` (type: `integer`):

Consecutive short sections on the same page are merged until they reach roughly this size. A 30-token chunk cannot answer a question — it only adds noise to retrieval. Set to 0 to keep every heading as its own chunk.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxPages": 100,
  "followLinks": true,
  "sameOriginOnly": true,
  "maxChunkChars": 1500,
  "minChunkChars": 120,
  "overlapChars": 100,
  "dropRepeatedBoilerplate": true,
  "nearDuplicateDetection": true,
  "incrementalMode": false,
  "changedOnly": false,
  "snapshotKey": "CORPUS_SNAPSHOT",
  "concurrency": 8,
  "maxItems": 20000,
  "targetChunkChars": 400
}
```

# Actor output Schema

## `results` (type: `string`):

Citable, AI-ready documentation chunks.

## `qualityReport` (type: `string`):

Pages crawled, boilerplate collapsed, token totals and change stats.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("alaudinburki/docs-to-answers").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }] }

# Run the Actor and wait for it to finish
run = client.actor("alaudinburki/docs-to-answers").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ]
}' |
apify call alaudinburki/docs-to-answers --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,alaudinburki/docs-to-answers"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6c6m4nkbAIcRF2B4K/builds/uYdAyIXuQ0krerRPB/openapi.json