Sitemap to Markdown Crawler — Website to Markdown for LLM & RAG

What does Sitemap to Markdown Crawler do?

This website to markdown crawler takes a domain, finds its robots.txt and sitemap.xml files, downloads each page over plain HTTP (no browser) and returns clean Markdown with title, description, headings, link counts, word count, lastmod and a content hash. Export as JSON, CSV, Excel, XML or HTML from the Apify dataset, or call it through the API.

It is built for feeding documentation sites, blogs and help centres into LLM / RAG pipelines, and for re-crawling only what changed.

Who is it for? Use cases

• RAG and vector databases: index a documentation site from its sitemap.
• Weekly knowledge-base refresh: turn on incremental mode, so unchanged pages are neither output nor billed.
• Support chatbots: convert a help centre to Markdown.
• Content audits: word counts, headings, meta descriptions and internal/external link counts per page.
• Blog export: Markdown for migration or LLM fine-tuning datasets.

Features

• Sitemap discovery: robots.txt → sitemap.xml, sitemap indexes and .xml.gz. Pass a sitemap URL directly if you prefer.
• Link-following fallback (depth 0–10) when no sitemap exists; or choose sitemap_only / links_only.
• HTTP only: no headless browser, so runs are cheap and fast. JavaScript-rendered pages with almost no server-side text (< 20 words) are skipped, not billed.
• Incremental runs: skipUnchanged stores a content hash per URL; changedSince filters on sitemap lastmod.
• Main-content extraction with main/article heuristics, or your own contentSelector; removeSelectors strips cookie banners and the like.
• URL include/exclude globs, optional cleaned HTML, robots.txt respected by default.
• Optional Apify Proxy if a site blocks direct requests.

Input

Field	Type	Default	Description
startUrls (required)	array	crawlee.dev	Domains, homepages, sitemap URLs or seed pages (max 100).
maxResults	integer	50	Max pages delivered and billed (1–100000).
discoveryMode	string	sitemap_then_links	sitemap_then_links, sitemap_only, links_only.
includeUrlPatterns / excludeUrlPatterns	string list	empty	Glob patterns, e.g. https://crawlee.dev/js/docs/**, **/*.pdf.
maxCrawlDepth	integer	3	Link depth when following links (0–10).
changedSince	string	–	ISO date; sitemap URLs with an older lastmod are skipped (URLs without lastmod are kept).
skipUnchanged	boolean	false	Skip pages whose content hash is unchanged since the last run with the same stateKey.
stateKey	string	first host	Name of the incremental state.
contentSelector	string	–	CSS selector of the main content.
removeSelectors	string list	empty	CSS selectors removed before conversion.
includeHtml	boolean	false	Also output cleaned HTML.
respectRobotsTxt	boolean	true	Skip URLs disallowed by robots.txt.
maxConcurrency	integer	10	Parallel requests (1–50).
proxyConfiguration	object	none	Apify Proxy settings.

Example input:

{
  "startUrls": [{ "url": "https://crawlee.dev" }],
  "maxResults": 50,
  "discoveryMode": "sitemap_then_links",
  "skipUnchanged": true
}

Output

Real item from a production run (Markdown shortened):

{
  "url": "https://crawlee.dev/blog/building-gradcracker-scraper-with-crawlee",
  "requestedUrl": "https://crawlee.dev/blog/building-gradcracker-scraper-with-crawlee",
  "domain": "crawlee.dev",
  "title": "Building a Gradcracker scraper with Crawlee: anti-bot failures and redirect bugs | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "A real-world breakdown of building a Gradcracker scraper with Crawlee, handling bot detection, unstable redirects, and production trade-offs.",
  "language": "en",
  "markdown": "I spent three days debugging a scraper that I really expected to have knocked out in an afternoon. ...",
  "html": null,
  "headings": [{ "level": 2, "text": "What JobOps does" }, { "level": 2, "text": "Conclusion" }],
  "wordCount": 1738,
  "links": { "internal": 5, "external": 14 },
  "canonicalUrl": "https://crawlee.dev/blog/building-gradcracker-scraper-with-crawlee",
  "lastmod": null,
  "lastModifiedHeader": "2026-10-02T13:44:42.000Z",
  "contentHash": "2b60f59248d46f98f513e382968ead77f29f5812060f8e9206ac94751f88aa70",
  "changeStatus": "unknown",
  "discoveredFrom": "sitemap",
  "httpStatus": 200,
  "crawledAt": "2026-10-04T08:22:59.686Z"
}

Output fields & fill rates

Measured on a 40-page production run (crawlee.dev):

Field	Fill rate	Notes
url, requestedUrl, domain, canonicalUrl	100%
title, markdown, headings, links, wordCount	100%
language, httpStatus, contentHash, crawledAt, discoveredFrom	100%
lastModifiedHeader	100%	From the HTTP Last-Modified header, on this site.
description	92.5%	Meta description, empty when the page has none.
lastmod	17.5%	Only when the sitemap provides it.
changeStatus	100%	new / changed when skipUnchanged is on (new seen in proof runs), otherwise unknown.
html	0%	Only filled with includeHtml: true.

Fill rates depend on the target site.

Examples of each mode (real proof runs on crawlee.dev)

Mode	Input	Result
Sitemap (default)	{"startUrls":[{"url":"https://crawlee.dev"}],"maxResults":10}	10 pages, discoveredFrom: "sitemap"
Incremental, first run	same input + "skipUnchanged": true, "stateKey": "qa-incremental"	10 pages, all changeStatus: "new" (run EbEsitDuybxtzu5TY)
Incremental, second run	identical input, same stateKey	10 different pages, changeStatus: "new". The pages delivered by the first run were fetched again (20 requests in total) but not output or billed, so a repeated run only returns pages not seen before (run aR6iZwob6L0O9C1X8).
Link following	{"startUrls":[{"url":"https://crawlee.dev/js/docs/quick-start"}],"discoveryMode":"links_only","maxCrawlDepth":1,"maxResults":10,"includeUrlPatterns":["https://crawlee.dev/js/docs/**"]}	10 pages, first one discoveredFrom: "start_url" (run B4i0MAl3Wt6b5BwgE)
Unreachable host	INPUT.edge-unreachable.json	0 items, no error row, run SUCCEEDED (run Iht73mb4UQbj6JKb2)

changeStatus is unknown when skipUnchanged is off.

How to use via API

curl

curl -X POST "https://api.apify.com/v2/acts/leadsbrary~sitemap-to-markdown-crawler/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://crawlee.dev"}],"maxResults":50}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('leadsbrary/sitemap-to-markdown-crawler').call({
  startUrls: [{ url: 'https://crawlee.dev' }],
  maxResults: 50,
  skipUnchanged: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].markdown);

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("leadsbrary/sitemap-to-markdown-crawler").call(run_input={
    "startUrls": [{"url": "https://crawlee.dev"}],
    "maxResults": 50,
    "skipUnchanged": True,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items[0]["markdown"])

Pricing

Pay per result: one result = one page delivered. Price per 1,000 pages depends on your Apify plan:

Plan	Per 1,000 results
Free	$1.20
Bronze	$1.04
Silver	$0.92
Gold and above	$0.80

Skipped pages (unchanged, near-empty, blocked, disallowed) are not billed. maxResults caps both output and cost. Apify platform usage is separate; the production runs used about $0.0005 of platform usage for 10–40 pages.

Troubleshooting

• 0 results: the site may have no reachable sitemap and link-following may be off (sitemap_only), or the site was unreachable. A test run against an unreachable host finished with 0 items and no error rows. Check the run log.
• Pages missing or near-empty: the site renders content with JavaScript. This Actor does not run a browser and skips pages with fewer than 20 words.
• Blocked (403/429): lower maxConcurrency and enable Apify Proxy in proxyConfiguration.
• Wrong content extracted: set contentSelector and removeSelectors.
• URLs skipped: check respectRobotsTxt and your include/exclude patterns.
• "startUrls is required": provide at least one URL.

FAQ

Does it run a browser? No, plain HTTP only, which keeps it cheap and fast.

How do I only get changed pages? Set skipUnchanged: true and reuse the same stateKey. Use changedSince to also filter on sitemap lastmod.

Can I crawl several sites in one run? Yes, up to 100 start URLs; maxResults is shared across them.

Is it legal to crawl a site? It only reads public pages and respects robots.txt by default. Check the target's terms of service and copyright rules for your use of the content.

Keywords

website to markdown crawler, sitemap to markdown, sitemap crawler, website to markdown, html to markdown, llm docs scraper, rag website crawler, documentation scraper, website content crawler, markdown scraper

Hashtags

#markdown #sitemap #crawler #llm #rag #webscraping #documentation #ai #apify