# Website to Markdown & RAG Chunks for AI (`belgair/website-markdown-rag`) Actor

Crawl public websites over HTTP and get clean Markdown plus ready-to-embed chunks with heading path, token estimate and hash. Sitemaps, robots.txt and a changed-pages mode. You only pay for pages that returned text.

- **URL**: https://apify.com/belgair/website-markdown-rag.md
- **Developed by:** [Belgair Data](https://apify.com/belgair) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.26 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown & RAG Chunks for AI

Crawl public web pages (documentation, help centers, blogs) over plain HTTP and get, per page, the clean Markdown of the main content plus a list of chunks with their heading path, token estimate and hash. Input: one or more start URLs. Output: one dataset row per page with text. You pay $1.80 per 1,000 pages with text; empty, blocked, failed, duplicate and unchanged pages are free.

### What can you use it for?

- **Build a RAG knowledge base**: crawl your documentation, help center or blog and load the chunks into a vector database. Each chunk has a stable `chunkId`, its `headingPath` and a token estimate, so you can upsert and cite without extra processing.
- **A chatbot on your own documentation**: give an assistant the exact text of your docs, split along the headings of each page, with the source URL on every chunk.
- **Keep an index up to date**: schedule a weekly run with "Only changed pages". You get only new pages and pages whose text changed, and you pay only for those.
- **Track text changes on a website**: compare `contentHash` across runs, or let the change mode do it; removed pages are listed in a separate record.
- **Text for LLM projects**: summaries, evaluation sets or content audits from Markdown without menus, cookie banners or footers.

### Example output

One row per page, from a real run on the Crawlee documentation (text shortened with "...", 2 of the 7 chunks shown):

````json
{
    "url": "https://crawlee.dev/js/docs/quick-start",
    "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
    "language": "en",
    "lastModified": "2026-10-15T15:24:34.000Z",
    "fetchedAt": "2026-10-19T07:16:43.256Z",
    "depth": 0,
    "contentSource": "main",
    "contentShare": 0.94,
    "tokenEstimate": 2476,
    "contentHash": "cb006dbc38523163cf5715acbcba199565239878172ee9a87497a8573892e675",
    "changeType": null,
    "chunkCount": 7,
    "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.\n\n## Choose your crawler\n\n...",
    "chunks": [
        {
            "chunkId": "222b8678e6fffec6-0",
            "chunkIndex": 0,
            "chunkCount": 7,
            "headingPath": ["Quick Start"],
            "chunkText": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. ...",
            "chunkTokens": 427,
            "chunkHash": "4c3921e6a7ea8d13cd72db4300e4b7efee817f80c75ab2dea2c467f60d4532a5",
            "url": "https://crawlee.dev/js/docs/quick-start",
            "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
            "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
            "language": "en",
            "lastModified": "2026-10-15T15:24:34.000Z",
            "fetchedAt": "2026-10-19T07:16:43.256Z"
        },
        {
            "chunkId": "222b8678e6fffec6-1",
            "chunkIndex": 1,
            "chunkCount": 7,
            "headingPath": ["Quick Start", "Installation with Crawlee CLI"],
            "chunkText": "## Installation with Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. ...\n\n```bash\nnpx crawlee create my-crawler\n```\n\n...",
            "chunkTokens": 242,
            "chunkHash": "3663e5b0a79f5b1ded09364d8097859142bba960a5f374622fe4f9cb15aa1c5c",
            "url": "https://crawlee.dev/js/docs/quick-start",
            "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
            "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
            "language": "en",
            "lastModified": "2026-10-15T15:24:34.000Z",
            "fetchedAt": "2026-10-19T07:16:43.256Z"
        }
    ]
}
````

That run took 11 seconds for 10 pages and 60 chunks.

#### Output fields (one row per page)

| Field | Type | What it is |
| --- | --- | --- |
| `url` | string | The page as fetched, after redirects, correctly encoded |
| `canonicalUrl` | string or null | The canonical address the page declares |
| `title` | string | The page title (`<title>`, else `og:title`, else the first heading) |
| `description` | string or null | The meta description (or `og:description`) |
| `language` | string or null | Language code the page declares (`en`, `de-ch`) |
| `lastModified` | string or null | ISO date: the Last-Modified header, else a date in the page (`article:modified_time`, `og:updated_time`, schema.org `dateModified`, or a "Last updated" date as on GitBook), else the sitemap date |
| `fetchedAt` | string | ISO date and time the page was fetched |
| `depth` | integer | Clicks away from a start URL (start URLs 0, sitemap pages 1) |
| `contentSource` | string | `main`, `article` or `body`: where the main content came from |
| `contentShare` | number | Share of the visible page text kept in the Markdown, 0 to 1. Lower values are normal on pages with large menus or repeated tab content |
| `tokenEstimate` | integer | Estimated tokens of the whole page (about 4 characters per token) |
| `contentHash` | string | SHA-256 of the clean Markdown with whitespace collapsed: same text, same hash |
| `changeType` | string or null | `new` or `changed` with "Only changed pages", otherwise null |
| `chunkCount` | integer | Number of chunks of the page |
| `markdown` | string | The full clean Markdown of the page |
| `chunks` | array | The chunks, in order (see below) |

Each item in `chunks` has `chunkId` (stable: the same page and index give the same id), `chunkIndex` (from 0), `chunkCount`, `headingPath` (the headings above the chunk, from the top), `chunkText` (Markdown), `chunkTokens`, `chunkHash` (SHA-256 of `chunkText` with whitespace collapsed), and the page fields `url`, `canonicalUrl`, `title`, `language`, `lastModified` and `fetchedAt`, so a chunk can be stored on its own.

Want one record per chunk? Open the **Chunks** view of the dataset, or add `unwind=chunks` to the dataset items API URL. The **Markdown** view shows the URL, title and Markdown only.

### Why this crawler

- **No surprise bills.** A fixed price per page with text. Empty, blocked, failed, duplicate and unchanged pages are free, and "Max pages" is a hard cap: the most a run can cost is Max pages × the page price, plus the start fee.
- **The same input gives the same pages.** A fixed order: the start URLs, then the links on those pages (links in the main text before menu links), then the sitemap, then deeper links. With a small "Max pages" you get the pages your start URL points to, not a random part of the site.
- **All of the main content.** Every `<main>` block, or every `<article>` when there is no `<main>`, with a fallback to the page body when the main block is too small. `contentShare` shows how much text was kept.
- **Clean Markdown.** Links on one line with absolute URLs, tables, nested lists, and code blocks with their language: read from the page (Sphinx, MkDocs Material, Docusaurus, MDN, GitHub), else a careful guess for clear cases (JSON, HTML/XML, Python, JavaScript/TypeScript, shell commands, TOML), else none. The content of every tab is kept, with the tab name above it. No images or base64 (alt text is kept), navigation, footers, cookie banners, tables of contents, "Was this helpful?" widgets, wiki edit links, maintenance banners or reference lists.
- **A change mode you can switch off.** It compares the clean text, not ETag headers. Lines that change every day without a real change ("Last updated 28 days ago") are left out of the text, so they never count as a change.
- **Clear errors.** Every failed page is counted by cause (rate limit, blocked, network, other). A blocked page is reported, never retried around.
- **Fast and light.** HTTP only, no browser: the first pages arrive within seconds, and 256 MB of memory is enough.

### How it works

1. **robots.txt first.** For every host it reads robots.txt and follows its rules for our crawler name (BelgairBot) or for all crawlers, including Crawl-delay. Without a robots.txt (404) everything is allowed; when robots.txt cannot be read (server error or no connection) nothing on that host is crawled. Pages marked `noindex` (meta robots or X-Robots-Tag) are not delivered, and links on `nofollow` pages are not followed.
2. **A calm pace.** At most about one request per second per host, slower when Crawl-delay asks for it. A host that asks for more than 60 seconds between requests is skipped and reported.
3. **Plain HTTP.** Pages are fetched as HTML without a browser, with an honest user agent and without cookies.
4. **Main content.** Scripts, styles, forms, hidden elements, navigation, sidebars, footers, cookie and consent banners, share buttons, author bylines and tables of contents are removed; the main blocks are converted to Markdown.
5. **Chunks.** A page is split at its headings. Sections are packed into chunks up to "Chunk size"; a section that is too long is split by paragraph, list item, table row, code line or sentence, so code blocks and tables stay valid Markdown. "Chunk overlap" repeats whole blocks from the previous chunk of the same section.
6. **Privacy.** E-mail addresses and phone numbers in the text are always replaced by `[email]` and `[phone]`, also inside code samples. Phone numbers are recognised in the usual ways of writing them per country (leading 0 or (0), country code, North American style, grouped with spaces, dots, dashes, slashes or brackets), while dates, times, versions, prices, postcodes and file names stay as they are. There is no setting to turn this off.

### Pricing

- **$1.80 per 1,000 pages with text** ($0.0018 per page) on the free Apify plan, plus the Actor start fee ($0.00005). All chunks of a page are included.
- On paid Apify plans: Bronze $1.62, Silver $1.44, Gold and higher $1.26 per 1,000 pages.
- Not charged: pages that are empty, empty without JavaScript, blocked, failed, not found, not HTML, duplicates, marked noindex, and in the change mode pages that did not change.
- The most a run can cost is "Max pages" × the page price. On the free plan 100 pages cost at most $0.18, 1,000 pages at most $1.80.

### Reliability

- Retries with a growing pause for network errors, server errors and rate limits (Retry-After is respected).
- A run stops cleanly before its timeout and keeps everything delivered so far.
- A run with no page with text fails with a clear reason instead of succeeding empty, so you notice it in your schedule.

### What it cannot do

- **Pages behind a login.** It only reads public pages and never logs in.
- **JavaScript-only sites.** Without a browser, a page that only shows content after JavaScript runs is empty. Such pages are counted as "empty without JavaScript" and not charged. Many of those sites still have a sitemap or server-rendered docs pages; try those as start URLs.
- **Blocked sites.** A site that answers with 401, 403 or a challenge page is reported and skipped. The crawler never solves CAPTCHAs and never works around blocks.
- **Files.** PDFs, images and other files are skipped (counted as "not HTML").

### Your responsibility

You choose the websites. You are responsible for following the terms of use of the sites you crawl and for how you use the text. The crawler follows robots.txt and Crawl-delay, keeps a calm pace and masks e-mail addresses and phone numbers, but that does not replace the permission of the site owner where it is needed. This Actor is not affiliated with or endorsed by the websites you crawl.

### Input

Only `startUrls` is required. Example:

```
{
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "scope": "folder",
    "maxResults": 10
}
```

| Field (JSON key) | Type | Default | Allowed | What it does |
| --- | --- | --- | --- | --- |
| Start URLs (`startUrls`) | array of `{ "url": "…" }` | | 1 to 100 | Pages to start from. A start URL is always fetched, even outside the path prefix; its links follow the scope |
| Scope (`scope`) | string | `folder` | `folder`, `host`, `domain` | Folder of the start URL, the whole host, or the host and its subdomains |
| Path prefix (`pathPrefix`) | string | | | Only paths that start with this, e.g. `/docs/v2/`. Replaces the folder of the start URL |
| Skip paths (`excludePaths`) | array of strings | | | Skip paths that start with one of these, e.g. `/blog/tag/`; `*` matches anything, e.g. `/*/print/` |
| Max pages (`maxResults`) | integer | 100 | 1 to 10,000 | Stop after this many pages with text; also the cost cap |
| Chunk size (`chunkSize`) | integer | 500 | 100 to 8,000 | Target size per chunk, in estimated tokens |
| Chunk overlap (`chunkOverlap`) | integer | 50 | 0 to 1,000 | Whole blocks repeated from the previous chunk of the same section, 0 = none |
| Only changed pages (`onlyChanged`) | boolean | false | | Only new and changed pages since the last run with the same start URLs, scope, path prefix, skip paths and link depth |
| Use sitemaps (`useSitemaps`) | boolean | true | | Read sitemaps from robots.txt or /sitemap.xml, including sitemap indexes |
| Extra sitemap URLs (`sitemapUrls`) | array of strings | | | Sitemaps to read as well |
| Max link depth (`maxDepth`) | integer | 10 | 0 to 50 | Clicks away from a start URL |

Folder scope: for `https://example.com/docs/intro` the crawler stays under `/docs/`. When a start URL redirects to another folder, that folder is added to the scope. To crawl a section such as `https://docs.example.com/academy`, end the start URL with a slash (`/academy/`) or set the path prefix `/academy/`.

### Run files in the key-value store

- **`RUN_SUMMARY`** (every run): `results` (pages delivered), `chunks`, `pagesOk`, `pagesFailed`, `failedRequestsByCategory` (proxy, rateLimit, blocked, network, other), `stopReason`, `durationSecs`, `scope`, `skippedUrls` and `skippedByReason` with `skippedExamples` (robots.txt, noindex, outside the scope, too deep, over the limit, not HTML, not found, empty without JavaScript, too little text, duplicate, unchanged, host skipped), `hosts` (robots.txt and Crawl-delay per host), `sitemaps` (files read, URLs found and in scope) and, in the change mode, `changes` (new, changed, unchanged, removed).
- **`REMOVED_PAGES`** (change mode only): `removedAt` and `urls` of pages that were known from the last run but are gone now, written only after a crawl that reached the end without failed pages.

### Schedule it: only changed pages

Turn on "Only changed pages" and schedule the task (for example weekly). The first run returns every page with `changeType: "new"`, so your index starts complete. Every later run fetches the pages again, compares the clean text and returns only pages that are new or changed. A run without changes costs only the start. "Max pages" counts delivered pages only: unchanged pages are checked for free and do not use up the limit, so a run with a small limit on a large site keeps adding pages it has not delivered yet. The hashes are kept in a named key-value store in your own account (`website-markdown-rag-changes`), one record per combination of start URLs, scope, path prefix, skip paths and link depth; it holds only those settings and page hashes.

### FAQ

**Does it render JavaScript?** No. It reads the HTML as the server sends it, which is fast and cheap. Pages that are empty without JavaScript are reported and free.

**Why did a run return fewer pages than Max pages?** The site has fewer pages in scope, or URLs were skipped. `RUN_SUMMARY` shows each reason with examples.

**How do I get one record per chunk?** Use the Chunks view, or `unwind=chunks` on the dataset items API.

**Can I crawl several sites in one run?** Yes, add several start URLs; the scope is applied to each of them.

### More from Belgair Data

- [European ATS Jobs](https://apify.com/belgair/european-ats-jobs): Open jobs from company career sites across Europe
- [UK Companies House Search by SIC Code & Postcode](https://apify.com/belgair/companies-house-uk): Search the UK company register by name, SIC code, postcode or town
- [French Company Search (SIRENE) by NAF & Department](https://apify.com/belgair/sirene-france): Search the French company register by NAF code, department and status

If this Actor saved you time, a short review on Apify Store helps other users find it.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start from, on public websites you are allowed to crawl. The crawler follows links and the sitemap from there, within the scope below. Up to 100 start URLs.

## `scope` (type: `string`):

Which links to follow. Folder (default): the same host and the folder of the start URL, so https://example.com/docs/intro crawls everything under /docs/. Host: the whole host. Domain: the host and its subdomains.

## `pathPrefix` (type: `string`):

Only crawl paths that start with this, for example /blog/ or /docs/v2/. Replaces the folder of the start URL. Leave empty to use the scope above.

## `excludePaths` (type: `array`):

Skip URLs whose path starts with one of these, for example /blog/tag/ or /docs/v1/. Use \* for any characters, for example /\*/print/. Skipped URLs count as outside the scope in RUN_SUMMARY and cost nothing.

## `maxResults` (type: `integer`):

Stop after this many pages with text. This is also the most you can be charged for in one run: empty, blocked, failed and duplicate pages are free.

## `chunkSize` (type: `integer`):

Target size of each chunk, in estimated tokens (about 4 characters per token). Chunks follow the headings of the page; a section that is too long is split by paragraph, list item, table row, code line or sentence.

## `chunkOverlap` (type: `integer`):

How much of the previous chunk is repeated at the start of the next one, within the same section. Whole blocks only, so code and tables stay intact. 0 = no overlap. At most half the chunk size.

## `onlyChanged` (type: `boolean`):

Return only pages that are new or whose clean text changed since the last run with the same start URLs, scope, path prefix, skip paths and link depth (each combination keeps its own memory). The first run returns every page. Pages that did not change are checked but not delivered and not charged. Off = every run returns all pages again.

## `useSitemaps` (type: `boolean`):

Read the sitemaps listed in robots.txt (or /sitemap.xml), including sitemap indexes, and queue the pages that fall within the scope. Order: the start URLs first, then the links on those pages (links in the main content before menu links), then the sitemap, then deeper links.

## `sitemapUrls` (type: `array`):

Sitemaps to read in addition to the ones found automatically, for example https://example.com/sitemap-docs.xml.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL the crawler goes. Start URLs are depth 0, pages from the sitemap depth 1.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "scope": "folder",
  "maxResults": 10,
  "chunkSize": 500,
  "chunkOverlap": 50,
  "onlyChanged": false,
  "useSitemaps": true,
  "maxDepth": 10
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/js/docs/quick-start"
        }
    ],
    "scope": "folder",
    "maxResults": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("belgair/website-markdown-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "scope": "folder",
    "maxResults": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("belgair/website-markdown-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "scope": "folder",
  "maxResults": 10
}' |
apify call belgair/website-markdown-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,belgair/website-markdown-rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qGwMn1UrCAvxnunMG/builds/5crhjPI7WKfkjzvaI/openapi.json