Website to Markdown & RAG Chunks for AI

Crawl public web pages (documentation, help centers, blogs) over plain HTTP and get, per page, the clean Markdown of the main content plus a list of chunks with their heading path, token estimate and hash. Input: one or more start URLs. Output: one dataset row per page with text. You pay $1.80 per 1,000 pages with text; empty, blocked, failed, duplicate and unchanged pages are free.

What can you use it for?

• Build a RAG knowledge base: crawl your documentation, help center or blog and load the chunks into a vector database. Each chunk has a stable chunkId, its headingPath and a token estimate, so you can upsert and cite without extra processing.
• A chatbot on your own documentation: give an assistant the exact text of your docs, split along the headings of each page, with the source URL on every chunk.
• Keep an index up to date: schedule a weekly run with "Only changed pages". You get only new pages and pages whose text changed, and you pay only for those.
• Track text changes on a website: compare contentHash across runs, or let the change mode do it; removed pages are listed in a separate record.
• Text for LLM projects: summaries, evaluation sets or content audits from Markdown without menus, cookie banners or footers.

Example output

One row per page, from a real run on the Crawlee documentation (text shortened with "...", 2 of the 7 chunks shown):

{
    "url": "https://crawlee.dev/js/docs/quick-start",
    "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
    "language": "en",
    "lastModified": "2026-10-15T15:24:34.000Z",
    "fetchedAt": "2026-10-19T07:16:43.256Z",
    "depth": 0,
    "contentSource": "main",
    "contentShare": 0.94,
    "tokenEstimate": 2476,
    "contentHash": "cb006dbc38523163cf5715acbcba199565239878172ee9a87497a8573892e675",
    "changeType": null,
    "chunkCount": 7,
    "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.\n\n## Choose your crawler\n\n...",
    "chunks": [
        {
            "chunkId": "222b8678e6fffec6-0",
            "chunkIndex": 0,
            "chunkCount": 7,
            "headingPath": ["Quick Start"],
            "chunkText": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. ...",
            "chunkTokens": 427,
            "chunkHash": "4c3921e6a7ea8d13cd72db4300e4b7efee817f80c75ab2dea2c467f60d4532a5",
            "url": "https://crawlee.dev/js/docs/quick-start",
            "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
            "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
            "language": "en",
            "lastModified": "2026-10-15T15:24:34.000Z",
            "fetchedAt": "2026-10-19T07:16:43.256Z"
        },
        {
            "chunkId": "222b8678e6fffec6-1",
            "chunkIndex": 1,
            "chunkCount": 7,
            "headingPath": ["Quick Start", "Installation with Crawlee CLI"],
            "chunkText": "## Installation with Crawlee CLI\n\nThe fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**. ...\n\n```bash\nnpx crawlee create my-crawler\n```\n\n...",
            "chunkTokens": 242,
            "chunkHash": "3663e5b0a79f5b1ded09364d8097859142bba960a5f374622fe4f9cb15aa1c5c",
            "url": "https://crawlee.dev/js/docs/quick-start",
            "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
            "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
            "language": "en",
            "lastModified": "2026-10-15T15:24:34.000Z",
            "fetchedAt": "2026-10-19T07:16:43.256Z"
        }
    ]
}

That run took 11 seconds for 10 pages and 60 chunks.

Output fields (one row per page)

Field	Type	What it is
url	string	The page as fetched, after redirects, correctly encoded
canonicalUrl	string or null	The canonical address the page declares
title	string	The page title (<title>, else og:title, else the first heading)
description	string or null	The meta description (or og:description)
language	string or null	Language code the page declares (en, de-ch)
lastModified	string or null	ISO date: the Last-Modified header, else a date in the page (article:modified_time, og:updated_time, schema.org dateModified, or a "Last updated" date as on GitBook), else the sitemap date
fetchedAt	string	ISO date and time the page was fetched
depth	integer	Clicks away from a start URL (start URLs 0, sitemap pages 1)
contentSource	string	main, article or body: where the main content came from
contentShare	number	Share of the visible page text kept in the Markdown, 0 to 1. Lower values are normal on pages with large menus or repeated tab content
tokenEstimate	integer	Estimated tokens of the whole page (about 4 characters per token)
contentHash	string	SHA-256 of the clean Markdown with whitespace collapsed: same text, same hash
changeType	string or null	new or changed with "Only changed pages", otherwise null
chunkCount	integer	Number of chunks of the page
markdown	string	The full clean Markdown of the page
chunks	array	The chunks, in order (see below)

Each item in chunks has chunkId (stable: the same page and index give the same id), chunkIndex (from 0), chunkCount, headingPath (the headings above the chunk, from the top), chunkText (Markdown), chunkTokens, chunkHash (SHA-256 of chunkText with whitespace collapsed), and the page fields url, canonicalUrl, title, language, lastModified and fetchedAt, so a chunk can be stored on its own.

Want one record per chunk? Open the Chunks view of the dataset, or add unwind=chunks to the dataset items API URL. The Markdown view shows the URL, title and Markdown only.

Why this crawler

• No surprise bills. A fixed price per page with text. Empty, blocked, failed, duplicate and unchanged pages are free, and "Max pages" is a hard cap: the most a run can cost is Max pages × the page price, plus the start fee.
• The same input gives the same pages. A fixed order: the start URLs, then the links on those pages (links in the main text before menu links), then the sitemap, then deeper links. With a small "Max pages" you get the pages your start URL points to, not a random part of the site.
• All of the main content. Every <main> block, or every <article> when there is no <main>, with a fallback to the page body when the main block is too small. contentShare shows how much text was kept.
• Clean Markdown. Links on one line with absolute URLs, tables, nested lists, and code blocks with their language: read from the page (Sphinx, MkDocs Material, Docusaurus, MDN, GitHub), else a careful guess for clear cases (JSON, HTML/XML, Python, JavaScript/TypeScript, shell commands, TOML), else none. The content of every tab is kept, with the tab name above it. No images or base64 (alt text is kept), navigation, footers, cookie banners, tables of contents, "Was this helpful?" widgets, wiki edit links, maintenance banners or reference lists.
• A change mode you can switch off. It compares the clean text, not ETag headers. Lines that change every day without a real change ("Last updated 28 days ago") are left out of the text, so they never count as a change.
• Clear errors. Every failed page is counted by cause (rate limit, blocked, network, other). A blocked page is reported, never retried around.
• Fast and light. HTTP only, no browser: the first pages arrive within seconds, and 256 MB of memory is enough.

How it works

1. robots.txt first. For every host it reads robots.txt and follows its rules for our crawler name (BelgairBot) or for all crawlers, including Crawl-delay. Without a robots.txt (404) everything is allowed; when robots.txt cannot be read (server error or no connection) nothing on that host is crawled. Pages marked noindex (meta robots or X-Robots-Tag) are not delivered, and links on nofollow pages are not followed.
2. A calm pace. At most about one request per second per host, slower when Crawl-delay asks for it. A host that asks for more than 60 seconds between requests is skipped and reported.
3. Plain HTTP. Pages are fetched as HTML without a browser, with an honest user agent and without cookies.
4. Main content. Scripts, styles, forms, hidden elements, navigation, sidebars, footers, cookie and consent banners, share buttons, author bylines and tables of contents are removed; the main blocks are converted to Markdown.
5. Chunks. A page is split at its headings. Sections are packed into chunks up to "Chunk size"; a section that is too long is split by paragraph, list item, table row, code line or sentence, so code blocks and tables stay valid Markdown. "Chunk overlap" repeats whole blocks from the previous chunk of the same section.
6. Privacy. E-mail addresses and phone numbers in the text are always replaced by [email] and [phone], also inside code samples. Phone numbers are recognised in the usual ways of writing them per country (leading 0 or (0), country code, North American style, grouped with spaces, dots, dashes, slashes or brackets), while dates, times, versions, prices, postcodes and file names stay as they are. There is no setting to turn this off.

Pricing

• $1.80 per 1,000 pages with text ($0.0018 per page) on the free Apify plan, plus the Actor start fee ($0.00005). All chunks of a page are included.
• On paid Apify plans: Bronze $1.62, Silver $1.44, Gold and higher $1.26 per 1,000 pages.
• Not charged: pages that are empty, empty without JavaScript, blocked, failed, not found, not HTML, duplicates, marked noindex, and in the change mode pages that did not change.
• The most a run can cost is "Max pages" × the page price. On the free plan 100 pages cost at most $0.18, 1,000 pages at most $1.80.

Reliability

• Retries with a growing pause for network errors, server errors and rate limits (Retry-After is respected).
• A run stops cleanly before its timeout and keeps everything delivered so far.
• A run with no page with text fails with a clear reason instead of succeeding empty, so you notice it in your schedule.

What it cannot do

• Pages behind a login. It only reads public pages and never logs in.
• JavaScript-only sites. Without a browser, a page that only shows content after JavaScript runs is empty. Such pages are counted as "empty without JavaScript" and not charged. Many of those sites still have a sitemap or server-rendered docs pages; try those as start URLs.
• Blocked sites. A site that answers with 401, 403 or a challenge page is reported and skipped. The crawler never solves CAPTCHAs and never works around blocks.
• Files. PDFs, images and other files are skipped (counted as "not HTML").

Your responsibility

You choose the websites. You are responsible for following the terms of use of the sites you crawl and for how you use the text. The crawler follows robots.txt and Crawl-delay, keeps a calm pace and masks e-mail addresses and phone numbers, but that does not replace the permission of the site owner where it is needed. This Actor is not affiliated with or endorsed by the websites you crawl.

Input

Only startUrls is required. Example:

{
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "scope": "folder",
    "maxResults": 10
}

Field (JSON key)	Type	Default	Allowed	What it does
Start URLs (startUrls)	array of { "url": "…" }		1 to 100	Pages to start from. A start URL is always fetched, even outside the path prefix; its links follow the scope
Scope (scope)	string	folder	folder, host, domain	Folder of the start URL, the whole host, or the host and its subdomains
Path prefix (pathPrefix)	string			Only paths that start with this, e.g. /docs/v2/. Replaces the folder of the start URL
Skip paths (excludePaths)	array of strings			Skip paths that start with one of these, e.g. /blog/tag/; * matches anything, e.g. /*/print/
Max pages (maxResults)	integer	100	1 to 10,000	Stop after this many pages with text; also the cost cap
Chunk size (chunkSize)	integer	500	100 to 8,000	Target size per chunk, in estimated tokens
Chunk overlap (chunkOverlap)	integer	50	0 to 1,000	Whole blocks repeated from the previous chunk of the same section, 0 = none
Only changed pages (onlyChanged)	boolean	false		Only new and changed pages since the last run with the same start URLs, scope, path prefix, skip paths and link depth
Use sitemaps (useSitemaps)	boolean	true		Read sitemaps from robots.txt or /sitemap.xml, including sitemap indexes
Extra sitemap URLs (sitemapUrls)	array of strings			Sitemaps to read as well
Max link depth (maxDepth)	integer	10	0 to 50	Clicks away from a start URL

Folder scope: for https://example.com/docs/intro the crawler stays under /docs/. When a start URL redirects to another folder, that folder is added to the scope. To crawl a section such as https://docs.example.com/academy, end the start URL with a slash (/academy/) or set the path prefix /academy/.

Run files in the key-value store

• RUN_SUMMARY (every run): results (pages delivered), chunks, pagesOk, pagesFailed, failedRequestsByCategory (proxy, rateLimit, blocked, network, other), stopReason, durationSecs, scope, skippedUrls and skippedByReason with skippedExamples (robots.txt, noindex, outside the scope, too deep, over the limit, not HTML, not found, empty without JavaScript, too little text, duplicate, unchanged, host skipped), hosts (robots.txt and Crawl-delay per host), sitemaps (files read, URLs found and in scope) and, in the change mode, changes (new, changed, unchanged, removed).
• REMOVED_PAGES (change mode only): removedAt and urls of pages that were known from the last run but are gone now, written only after a crawl that reached the end without failed pages.

Schedule it: only changed pages

Turn on "Only changed pages" and schedule the task (for example weekly). The first run returns every page with changeType: "new", so your index starts complete. Every later run fetches the pages again, compares the clean text and returns only pages that are new or changed. A run without changes costs only the start. "Max pages" counts delivered pages only: unchanged pages are checked for free and do not use up the limit, so a run with a small limit on a large site keeps adding pages it has not delivered yet. The hashes are kept in a named key-value store in your own account (website-markdown-rag-changes), one record per combination of start URLs, scope, path prefix, skip paths and link depth; it holds only those settings and page hashes.

FAQ

Does it render JavaScript? No. It reads the HTML as the server sends it, which is fast and cheap. Pages that are empty without JavaScript are reported and free.

Why did a run return fewer pages than Max pages? The site has fewer pages in scope, or URLs were skipped. RUN_SUMMARY shows each reason with examples.

How do I get one record per chunk? Use the Chunks view, or unwind=chunks on the dataset items API.

Can I crawl several sites in one run? Yes, add several start URLs; the scope is applied to each of them.

More from Belgair Data

• European ATS Jobs: Open jobs from company career sites across Europe
• UK Companies House Search by SIC Code & Postcode: Search the UK company register by name, SIC code, postcode or town
• French Company Search (SIRENE) by NAF & Department: Search the French company register by NAF code, department and status

If this Actor saved you time, a short review on Apify Store helps other users find it.