Website to Markdown Crawler for RAG: Fast, No Browser

Give it a website, get one row per page with clean Markdown of the main content, plain text on request, headings, link counts and optional heading-based chunks with token estimates. It uses plain HTTP only: no browser, no proxy, no LLM. It reads the sitemap first, honors robots.txt including Crawl-delay, and has a changed-pages-only mode so a nightly refresh pays only for pages that changed.

USD 1.20 per 1,000 pages, plus a tiny start fee per run. Pages that fail, need JavaScript, are blocked by robots.txt, or are unchanged in only-changed mode are free.

What people use it for

• A RAG knowledge base that stays in sync. Documentation sites, help centres and engineering blogs become Markdown with real headings, lists, tables and code blocks. Turn on chunkMarkdown for chunks that carry their heading path and a token estimate, then embed them. Schedule it with onlyChanged and a refresh costs a few cents.
• Agent and MCP context. "Read this whole docs site" as one cheap, deterministic run that returns structured JSON, with no browser to start and nothing to babysit.
• Content and SEO exports. Export a site's text for audits, migrations, content gap analysis and translation memory. Each row has the title, description, canonical URL, language, headings outline, internal and external link counts and the schema.org types found on the page.
• Change tracking. contentHash and changed tell you which pages moved since the last run, and the run summary lists the URLs that vanished from the sitemap.

Sample output

Real run on a public documentation page (https://crawlee.dev/js/docs/introduction) with chunkMarkdown on; markdown, headings and chunks shortened:

{
  "url": "https://crawlee.dev/js/docs/introduction",
  "finalUrl": "https://crawlee.dev/js/docs/introduction",
  "httpStatus": 200,
  "depth": 0,
  "title": "Introduction | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "Your first steps into the world of scraping with Crawlee",
  "canonicalUrl": "https://crawlee.dev/js/docs/introduction",
  "language": "en",
  "markdown": "# Introduction\n\nCrawlee covers your crawling and scraping end-to-end and helps you **build reliable scrapers. Fast.**\n\nYour crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you...",
  "wordCount": 289,
  "tokenEstimate": 529,
  "headings": [
    { "level": 1, "text": "Introduction" },
    { "level": 2, "text": "What you will learn" },
    { "level": 2, "text": "Next steps" }
  ],
  "chunks": [
    { "index": 0, "headingPath": ["Introduction"], "markdown": "# Introduction\n\nCrawlee covers your crawling and scraping end-to-end and helps you **build...", "tokenEstimate": 143 },
    { "index": 1, "headingPath": ["Introduction", "What you will learn"], "markdown": "## What you will learn\n\nThe goal of the introduction is to provide a step-by-step guide to...", "tokenEstimate": 80 }
  ],
  "internalLinkCount": 0,
  "externalLinkCount": 0,
  "jsonLdTypes": ["BreadcrumbList", "ListItem"],
  "lastModified": "2026-10-06T14:19:18.000Z",
  "contentHash": "90161d458bbd9ca6f138bacc59268ef5643a1d67533e47cd34ba5a90e51d2aa2",
  "changed": null,
  "needsJavaScript": false,
  "skippedReason": null,
  "robotsAllowed": true,
  "extractor": "page",
  "error": null,
  "errorCode": null,
  "fetchedAt": "2026-10-06T20:06:26.456Z"
}

Code blocks keep every line and their language, and nested lists keep their structure. This is the Markdown of a real tutorial page on go.dev (excerpt):

## Write some code

Get started with Hello, World.

1. Open a command prompt and cd to your home directory.

   On Linux or Mac:

   ```
   cd
   ```

   On Windows:

   ```
   cd %HOMEPATH%
   ```

2. Create a hello directory for your first Go source code.

   For example, use the following commands:

   ```
   mkdir hello
   cd hello
   ```

Tables become pipe tables, with the header row first and | inside a cell escaped. This one is from a real documentation page for Python's json module:

| JSON | Python |
| --- | --- |
| object | dict |
| array | list |
| string | str |
| number (int) | int |
| number (real) | float |
| true | True |
| false | False |
| null | None |

Quick start

The default input crawls 25 pages of the public Crawlee documentation in about 30 seconds. To crawl your own site:

{
  "startUrls": [{ "url": "https://example.com/docs/" }],
  "crawlScope": "sitemap_then_links",
  "maxPages": 200,
  "chunkMarkdown": true
}

From Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("JTPalms/website-to-markdown-crawler").call(run_input={
    "startUrls": [{"url": "https://example.com/docs/"}],
    "maxPages": 200,
    "chunkMarkdown": True,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(page["url"], page["wordCount"], page["changed"])

How it finds pages

crawlScope	What happens
sitemap_then_links (default)	Reads the sitemap (robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml), expands sitemap indexes and .gz files. If the sitemap lists fewer pages than maxPages, it also follows links from the pages it fetched, up to maxDepth.
sitemap	Sitemap pages and your start URLs only.
links_only	Breadth-first from your start URLs, same site only, up to maxDepth.
single_pages	Exactly the URLs you list.

• Scope. Pages must be on the same host as a start URL (www. makes no difference). A start URL with a path, for example https://example.com/docs/intro, keeps the crawl inside its folder (/docs/); turn off stayInStartFolder to crawl the whole site. To crawl another host, add it as a start URL.
• Nearest first. Sitemap pages are queued by how many leading path segments they share with your start URL, so a small maxPages gets the pages around the one you named.
• Include and exclude patterns. includeUrlPatterns and excludeUrlPatterns take three kinds of pattern, tested against the full URL (and the URL without its query): a glob where * matches inside one path segment and ** across segments (**/docs/**); plain text that the URL must contain (/blog/); or a regular expression written as re:^https://example\.com/(en|de)/. The default excludes skip author, team, people, staff and contact pages, which are mostly personal details. Pages you list as start URLs are always crawled.
• Links that are not followed. Links marked rel="nofollow" and links on a page whose robots meta says nofollow are not followed. Sitemap URLs and start URLs are still read.
• Duplicates. #fragment variants and tracking parameters (utm_*, gclid, fbclid and similar) are removed before a URL is queued. A page whose canonical link points to a page already delivered (a trailing-slash twin, a ?ref= variant, a redirect to a page you already have) comes back as a free row with skippedReason: "duplicate". If a page claims a canonical but its content differs, it is still delivered, so a misconfigured canonical cannot hide real pages.
• Other files. PDF links are reported once with skippedReason: "pdf" and are never downloaded (use PDF to Markdown and Text, JTPalms/pdf-to-markdown, for those). Images, archives and other non-page files are ignored. A URL that answers with a non-HTML content type is skipped from its headers alone.

Input

Field	What it does
startUrls	Pages or site roots, one per line.
crawlScope	sitemap_then_links (default), sitemap, links_only, single_pages.
maxPages	Hard cap on pages requested in the run, across all sites. Default 100, prefill 25, up to 50,000. Pages that fail still count, because they were requested.
maxDepth	Link depth when following links. Default 3. Start URLs and sitemap pages are depth 0.
stayInStartFolder	On by default. See scope above.
includeUrlPatterns, excludeUrlPatterns	See above.
contentMode	page (default): the cleaned body or <main>, best for documentation. article: Mozilla Readability's main article, best for blogs and news; falls back to page when no article is found (extractor says which one was used).
outputFormat	markdown (default) or markdown_and_text (adds text, the same reading order with no Markdown syntax).
chunkMarkdown, maxChunkTokens	Add chunks: the Markdown split at headings into pieces of at most maxChunkTokens estimated tokens (default 500, 50 to 4,000).
includeImages	Keep images as . Off by default.
includeLinks	Add links: the internal and external links in the page content, deduplicated, with link text and an internal flag. The counts are always there.
onlyChanged	Only output pages that are new or changed since your last only-changed run on the same site. See below.
useLlmsTxt	Also seed from the site's /llms.txt.
minWords	A page with fewer words is not charged. Default 30.
requestsPerSecondPerHost	Default 1, from 0.1 up to 2. Never more than 2 requests in flight and never closer than 500 ms to one host.
ignoreNoindex	Include pages marked noindex (they are skipped by default).
maxSitemapUrls	Most URLs read from a site's sitemaps. Default 5,000.

Output fields

Field	Meaning
url, finalUrl, httpStatus	The request, where it ended after redirects, and the final status.
depth	0 for start URLs and sitemap pages, link depth otherwise.
title, description, canonicalUrl, language	From the page: <title> (or og:title, or the first heading), meta description, rel=canonical made absolute, and the lang attribute.
markdown	The main content as Markdown: headings, lists, tables, code blocks with language, links made absolute, emphasis.
text	With markdown_and_text: plain text.
wordCount, tokenEstimate	Words in the content (code included; Chinese, Japanese and Korean count two characters per word), and an estimate of the tokens (characters divided by 4, labelled an estimate, not a tokenizer count).
headings	[{ level, text }], the outline of the content.
chunks	With chunkMarkdown: [{ index, headingPath, markdown, tokenEstimate }].
links	With includeLinks: [{ url, text, internal }].
internalLinkCount, externalLinkCount	Distinct links in the content, same site or not.
jsonLdTypes	schema.org types found in the page's JSON-LD (Article, FAQPage, BreadcrumbList and so on).
lastModified	The Last-Modified header, else the sitemap's lastmod, else null.
contentHash	SHA-256 of the Markdown.
changed	With onlyChanged: new or changed (unchanged pages are not output). Otherwise null.
needsJavaScript	true for an empty app shell (see below).
skippedReason	Why a row has no content, or null. See below.
robotsAllowed	false when robots.txt disallows the URL.
extractor	page, article, or article_fallback_page.
error, errorCode	Why the fetch failed, otherwise null.
fetchedAt	When the page was fetched.

The run's key-value store has a SUMMARY record: pages requested and billed, unchanged pages, skips by reason, errors, URLs filtered out, the sitemaps read, deletedUrls, requests and retries, and why the run stopped.

Rows without content (all free)

skippedReason or errorCode	Meaning
robots	robots.txt disallows the URL for this actor (robotsAllowed: false). It is never requested. Only start URLs get a row; disallowed links and sitemap entries are counted in the summary.
pdf, not_html	A PDF or another non-HTML file. The body is not downloaded.
noindex	The page asks not to be indexed (meta robots or X-Robots-Tag). Use ignoreNoindex to include it. Its links are still followed unless it says nofollow.
needs_javascript	Fewer than minWords words and the HTML is an app shell (empty <div id="root">, __NEXT_DATA__, "enable JavaScript"). needsJavaScript is true and markdown is empty.
below_min_words	A real page with too little text. The Markdown is delivered, but it is not charged.
duplicate	The same canonical page was already delivered.
redirect_off_site	A link redirects outside the crawl scope.
HTTP_ERROR, ACCESS_DENIED, RATE_LIMITED	HTTP 404, HTTP 403 (access denied), HTTP 429.
ROBOTS_UNREADABLE, CRAWL_DELAY_TOO_LONG	robots.txt could not be read (a server error), or it asks for a Crawl-delay over 60 seconds. The site is not crawled. One row per site says why.
TOO_LARGE, TIMEOUT, DNS, TLS, CONNECTION_REFUSED, PRIVATE_ADDRESS, BAD_REDIRECT, FETCH_FAILED, INTERNAL, INVALID_URL	Fetch problems and bad input, each with a plain message.

Refresh a knowledge base nightly (onlyChanged)

1. Create a task with your start URLs, crawlScope: "sitemap" and onlyChanged: true.
2. Run it once. Every page comes back as changed: "new". Load the rows or the chunks into your vector database, keyed by url, with contentHash as the version.
3. Add an Apify schedule, for example 0 3 * * *. From then on a run outputs only pages that are new or changed, and you pay only for those. A run with nothing changed outputs no rows and costs the start fee.
4. After each run read the SUMMARY record: deletedUrls lists URLs that were in your last run but are no longer in the sitemap, or that now answer 404 or 410. Remove them from your index.
5. Send the dataset to your vector database with the integration of your choice (Apify Console lists integrations on the task) or a webhook to your own loader.

How it works: state is kept per site in the named key-value store website-to-markdown-crawler-state (a short hash, Last-Modified and ETag per path). Sitemap and list runs send conditional requests, so a server that supports them answers 304 and nothing is downloaded. Changing contentMode, includeImages or any option that alters the Markdown makes pages look changed once. A page that prints a changing date or counter in its content counts as changed on every run. In links_only runs conditional requests are not used (a 304 carries no links to follow), but unchanged pages are still detected by their hash and not charged. Deleted-URL detection runs for sitemap-driven runs only.

Pricing

What	Price
One page converted (at least minWords words of Markdown)	USD 0.0012 (USD 1.20 per 1,000)
Run start	USD 0.001 per run
Failed, skipped, blocked, duplicate, JavaScript-only and unchanged pages	Free

Worked examples on the free plan:

• The default input (25 pages of documentation): 25 x 0.0012 + 0.001 = USD 0.031.
• A 500-page documentation site, first full run: 500 x 0.0012 + 0.001 = USD 0.601.
• The same site refreshed nightly with 10 changed pages: 10 x 0.0012 + 0.001 = USD 0.013 a night, about USD 0.39 a month.
• 20,000 pages across several sites, of which 1,500 fail or need JavaScript: 18,500 x 0.0012 = USD 22.20.

Higher Apify plans get a discount on the page price (10% on Silver, 20% on Gold and above). You can set a maximum cost per run: the actor stops cleanly when the budget is used and keeps every row already written. There is no proxy cost and no browser compute: this actor sends plain requests from the platform's own network.

Speed and limits

• Politeness sets the pace. One request per second per site by default (up to 2). On one site, 1,000 pages take about 17 minutes at the default; sites on different hosts are crawled in parallel. The default input (25 pages on one site) takes about 30 seconds.
• Work per page is small. Parsing, Markdown conversion and chunking take 10 to 30 milliseconds for a typical documentation page (3,000 pages of 72 KB each took 36 seconds of CPU on a laptop, 360 MB peak memory), so the wait is the politeness spacing, not the processing.
• Run timeout. The default run timeout is 1 hour. The actor stops about 25 seconds before the platform timeout so the summary and the changed-pages state are saved. Raise the run timeout for big crawls.
• Page size. Pages over 5 MB are skipped with TOO_LARGE. A row must fit in one Apify dataset item (under 9 MB): a larger row loses chunks and links first and then has its text cut at a paragraph (truncated: true, with a note).
• Memory. The default is 1 GB; a few pages are held in memory at once.

What it does not do

• No JavaScript. This is not a browser. A page that builds its content in the browser comes back flagged needsJavaScript (and is free) instead of empty and charged. Pages that load part of their content late (tabs, "load more" buttons, content fetched after the page loads) return only what the server sent. For those sites use a browser-based crawler such as Apify's Website Content Crawler, which is free apart from compute but heavier and slower.
• No logins, no paywall bypass, no CAPTCHA, no proxies, no fingerprint tricks. A site that answers an honest crawler with 403 or 429 gets a polite stop (see below), not an arms race. Sites behind bot protection such as Cloudflare often return 403 to any automated client.
• Content extraction is a heuristic. page mode removes navigation, headers, footers, sidebars, breadcrumbs, cookie banners, share bars, comments, scripts, hidden elements and permalink anchors, by tag, ARIA role, and common class and id words. A site that hides furniture with unusual class names can leak a menu into the Markdown, and a site that puts real content in an element called "sidebar" can lose it. Check a sample of your site. article mode suits articles and drops tables and short sections more often.
• Hidden tab panels are not read. Elements with the hidden attribute or display:none in the markup are treated as hidden and dropped.
• Tables become pipe tables when they have regular rows. Tables used for page layout are read as ordinary content. Cells that hold lists or code are flattened to one line with <br>.
• Languages. Any language works; the language of a page is the lang it declares.
• Email addresses and phone numbers are not extracted as fields and mailto: and tel: links are reduced to their text. The Markdown is the text of the page as published. The default excludes skip author, team, people, staff and contact pages.

Rights, robots.txt and politeness

• You are responsible for having the right to copy a site's content. This actor fetches public pages at addresses you supply. It does not republish anything itself. Check the site's terms and your own use case before you build a product on copied content.
• robots.txt is respected for every host, including after redirects. The User-agent: * rules apply to this actor, and so do rules written for the name website-to-markdown-crawler. Crawl-delay is honored (one request at a time, spaced as asked) up to 60 seconds; a longer delay means the site is skipped with a clear row. If robots.txt cannot be read because of a server error, the site is not crawled (a missing robots.txt, HTTP 404, means no rules). A disallowed URL is never requested and never retried.
• Honest identification. Every request uses the User-Agent Mozilla/5.0 (compatible; website-to-markdown-crawler/1.0; +https://apify.com/JTPalms), plain headers, no cookies, no proxies, no browser automation.
• Gentle by design. At most 2 requests in flight to one host, at least 500 ms between request starts (1 second by default), retries with back-off for temporary errors, and a stop after 5 consecutive 403 or 429 answers from one site. The crawl never goes outside your start hosts.
• Private and internal addresses (localhost, 10.x, 192.168.x, cloud metadata addresses) are refused.
• Pages marked noindex are skipped unless you ask for them.

Related actors

• PDF to Markdown and Text: Tables, Pages, Metadata for RAG: PDF parser and text extractor: turn PDF URLs into clean Markdown, plain text, per-page text and detected tables.
• Sitemap URL Extractor: All Pages of Any Website: Get every URL of a website from its XML sitemaps.
• News Article Extractor: Text, Date, Image from URLs and Feeds: News article scraper and extractor: turn article URLs, RSS or Atom feeds and news sitemaps into clean rows: title, Markdown and plain...
• SEO Audit & AI Search Readiness Checker (GPTBot, llms.txt): Crawl a site and audit every page for SEO basics (title, meta, H1, canonical, hreflang, noindex, alt text, JSON-LD, status, speed) plus...

FAQ

Why was a page not charged? Fetch errors, skipped pages (robots.txt, PDFs, noindex, duplicates), JavaScript-only shells, pages under minWords, and unchanged pages in onlyChanged mode are free. The row says why in skippedReason or error.

Why did I get fewer pages than maxPages? The sitemap and the links gave fewer pages in your scope, or the rest were filtered (check urlsFilteredOut in the summary), or the run hit its time or cost limit (also in the summary).

Can I crawl a site that needs a login? No. This actor reads public pages only.

Does the title contain the site name? title is the page's <title> as published. In article mode the Markdown starts with the page's own first heading when there is one.

How are token counts calculated? Characters divided by 4, rounded up. It is a quick estimate for planning chunk sizes, not the count of a particular model's tokenizer.