Website to Markdown & Sitemap to Markdown: RAG, llms.txt

What does Sitemap to Markdown do?

Sitemap to Markdown converter that turns any website into clean, LLM-ready Markdown using its sitemap (website to Markdown). Give it a domain like https://docs.example.com and it finds the sitemap on its own (robots.txt, /sitemap.xml, sitemap indexes, .xml.gz), filters the URLs you care about, downloads each page, strips the navigation, footers, cookie banners and scripts, and returns the main content as Markdown, with title, description, headings, word count, token estimate and optional RAG chunks. It can also build an llms.txt and llms-full.txt for the site, or act as a plain sitemap scraper that lists URLs with lastmod.

What makes it different

• Incremental sync that also tells you what was removed. With incremental on, unchanged pages are not fetched or charged, new and changed pages come back with changeStatus, and a CHANGES record lists new, changed, unchanged and removed URLs, so you can delete stale vectors too. Test: once all 36 htmx.org posts were saved, the next run returned 0 pages (36 unchanged).
• RAG chunks and llms.txt in the same run, at no extra charge. Heading-aware chunks with token estimates sit inside each page record, and llms.txt / llms-full.txt are written to the key-value store.
• Flat $0.50 per 1,000 pages. Plain HTTP at 256 MB, no browser and no proxy costs on top.

Real output from a test run (https://htmx.org/posts/, incremental, 300-token chunks, Oct 18 2026; text shortened):

{
  "url": "https://htmx.org/posts/2023-11-03-htmx-1-9-7-is-released/",
  "changeStatus": "new",
  "contentHash": "568bcacf33727b1eb3a36b522e2eb99d2cd8cfcbb034de323c72ec04975327d0",
  "title": "</> htmx ~ htmx 1.9.7 has been released!",
  "lastmod": "2023-11-03",
  "wordCount": 145,
  "tokenEstimate": 246,
  "source": "sitemap",
  "chunkCount": 1,
  "chunks": [
    {
      "index": 0,
      "heading": "htmx 1.9.7 has been released!",
      "text": "# htmx 1.9.7 has been released!\n\n## htmx 1.9.7 Release\n\nI’m happy to announce the [1.9.7 r …",
      "charCount": 982,
      "tokenEstimate": 246
    }
  ]
}

No sitemap? It crawls the site's links instead. Re-running on a schedule? Incremental mode only fetches and returns pages that are new or changed since the last run, and lists removed pages, so you only pay for what changed. Every page gets a contentHash, and exact or canonical duplicates are skipped.

No browser, no proxies, no login: plain HTTP, so it's fast and cheap. $0.50 per 1,000 pages.

Works on any website with a sitemap. Every example below was test-run and works as is: paste the JSON into the Input tab (switch to JSON view) and click Start.

Example use cases

1. Turn a docs site into Markdown chunks for a RAG chatbot

{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20,
  "chunkSize": 500
}

2. Keep a vector store in sync: only new and changed pages

Run it on a schedule with the same input. The first run saves a baseline; later runs skip pages whose sitemap lastmod hasn't changed (not fetched, not charged), drop pages whose content hash is the same, and list removed URLs in the CHANGES record.

{
  "startUrls": [
    "https://htmx.org/posts/"
  ],
  "maxPages": 0,
  "incremental": true
}

In our test the first run returned 36 pages (changeStatus: "new"); the second run returned 0 pages and reported 36 unchanged.

3. A site with no sitemap

{
  "startUrls": [
    "https://www.paulgraham.com"
  ],
  "maxPages": 25
}

The site has no sitemap, so the Actor crawls same-site links from the start URL (2 clicks deep by default). Each page says "source": "linkCrawl" and its crawlDepth. /index.html is skipped as a duplicate of /.

What data can Sitemap to Markdown extract?

Field	What it is
url / title	Page URL and title
markdown	Main page content as clean Markdown
lastmod	Last-modified date from the sitemap
contentHash	SHA-256 of the page's Markdown, for dedupe and change detection
changeStatus	new or changed (incremental mode only)
source / crawlDepth	Found in the sitemap or by linkCrawl, and how many clicks from the start URL
canonicalUrl	The page's rel=canonical URL
wordCount / tokenEstimate	Size of the page
chunks	RAG-ready chunks, when chunking is on
language	Page language

Sample output

Real rows from a run with the default input (text shortened):

url	title	wordCount	tokenEstimate	contentHash	source
https://docs.apify.com/academy	Apify Academy | Academy | Apify Documentation	215	460	4807eb7e09…	sitemap
https://docs.apify.com/academy/actor-marketing-playbook/acto…	Importance of Actor URL | Academy | Apify Documentation	1152	1672	848bcee4d3…	sitemap
https://docs.apify.com/academy/actor-marketing-playbook/acto…	Actor description & SEO description | Academy | Apify Documentation	1237	2008	bef532f82c…	sitemap

You can download the dataset as JSON, CSV, Excel, XML or HTML.

Front matter and thin-page flags

Each page's markdown starts with a YAML front matter block (turn off with Add YAML front matter):

---
title: "Actors and emojis | Academy | Apify Documentation"
url: "https://docs.apify.com/academy/..."
canonical_url: "https://docs.apify.com/academy/..."
lastmod: "2026-10-01"
language: "en"
word_count: 554
thin_content: false
scraped_at: "2026-10-18T07:40:00.000Z"
---

Use lastmod to re-pull only changed pages and canonical_url to skip duplicate paths. Pages with less main content than Thin page threshold (default 50 words) come back as type: "thin_page" with thinContent: true and are left out of llms.txt, so they don't end up in your embeddings as noise. Turn off Save thin pages to skip them entirely, and you won't be charged for them.

Incremental mode, link crawl and duplicates

• Incremental mode (incremental: true): the Actor remembers each page's sitemap lastmod and content hash in the named key-value store sitemap-to-markdown-state (one record per set of start URLs; set Incremental state name to keep separate histories). On the next run, pages with the same lastmod are skipped without being fetched, and fetched pages with the same hash are left out of the dataset (turn on Also output unchanged pages to keep them). You're charged only for pages in the dataset. The CHANGES record in the run's key-value store lists new, changed, unchanged and removed URLs, so you can upsert and delete vectors. Removed pages are detected when the whole sitemap was read (set Max pages to 0 or above the site's size, and leave Changed after empty). In this mode Max pages counts pages that need fetching, so unchanged pages don't use it up, and llms.txt covers only the pages returned in that run.
• Link crawl (linkCrawl): fallback (default) crawls same-site links from the start URL when there is no sitemap, or the sitemap has nothing under your start URL; always adds linked pages to the sitemap's pages; never turns it off. Depth is set by Max link-crawl depth (default 2). Links stay on the same site and under the start path, and follow your include/exclude patterns and robots.txt. Changed after only filters sitemap URLs, since crawled pages have no lastmod.
• Duplicates (skipDuplicates, on by default): a page whose Markdown is identical to one already saved in the run, or that shares a canonical URL with a saved page and has near-identical length, is skipped and not charged.
• Blocked sites: a 401/403/429 or a Cloudflare challenge comes back as a BLOCKED error record. This Actor uses plain HTTP without proxies or a browser, so it can't get past bot walls.

How to scrape a website to Markdown

1. Open the Input tab and enter a site or sitemap URL, for example https://docs.apify.com/academy.
2. Set Max pages, optional URL include/exclude patterns, and a chunk size if you want RAG chunks.
3. Click Start.
4. Download the pages as JSON, CSV or Markdown, or grab llms.txt and llms-full.txt from the key-value store.

You can also run it on a schedule, call it from the API tab, or connect it to Make, Zapier, n8n, Google Sheets or an AI agent through the Apify MCP server.

Who it's for

• RAG pipelines and vector databases: feed docs, help centers, blogs and knowledge bases into Pinecone, Weaviate, Qdrant, pgvector, Chroma or LlamaIndex/LangChain loaders.
• AI agents and MCP clients: give an agent a website's full content as Markdown in one call.
• Custom GPTs, Claude Projects and NotebookLM: download llms-full.txt and upload one file.
• llms.txt generation: publish an llms.txt index for your own site.
• Content audits and SEO: list every URL in a sitemap with lastmod (sitemap-only mode), word counts and headings.
• Incremental refreshes: schedule it with Incremental mode and only new or changed pages come back (and get charged).

Features

• Sitemap auto-discovery: robots.txt Sitemap: lines, /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz; nested sitemap indexes; gzip; plain-text sitemaps.
• Link-crawl fallback for sites without a usable sitemap (same-site, depth-limited).
• Incremental mode: skip pages unchanged since the last run by sitemap lastmod or content hash; changeStatus per page and a CHANGES record with removed URLs.
• Dedupe: contentHash on every page; identical and same-canonical duplicates skipped.
• Section scoping: https://example.com/blog keeps only pages under /blog.
• Filters: include/exclude URL patterns (glob like **/docs/** or /regex/), lastmod after a date or "30 days", newest-first ordering, max pages per site.
• Main-content extraction: semantic <main>/<article> detection with boilerplate removal, Mozilla Readability fallback, or your own CSS selector.
• Clean Markdown: ATX headings, fenced code blocks with language, GitHub tables, absolute links, no images by default (fewer tokens), heading permalink icons removed.
• RAG chunking: split by headings and paragraphs to a token or character size, with overlap and the heading path for each chunk.
• YAML front matter on every page: title, url, canonical_url, lastmod (from the sitemap), language, word_count, thin_content and scraped_at, so your pipeline can re-pull only changed pages and skip duplicate paths. Also available as the frontMatter field. Turn off with Add YAML front matter.
• Thin-page flag: pages with less main content than Thin page threshold (default 50 words) come back as type: "thin_page" with thinContent: true and are left out of llms.txt, so near-empty pages don't end up embedded as noise. Turn off Save thin pages to skip them entirely and not pay for them.
• llms.txt + llms-full.txt saved to the key-value store.
• Polite: respects robots.txt Disallow and Crawl-delay, concurrency limit, retries with backoff on 429/5xx.
• Clear error records: NO_SITEMAP_FOUND, BLOCKED, FETCH_FAILED, NOT_HTML, INVALID_URL.
• Any language: tested on Japanese and German sites; Chinese, Japanese and Korean text is counted per character for word counts, so CJK pages aren't wrongly flagged as thin.
• Spending-limit aware: stops cleanly at your run's max charge.

Input example

{
  "startUrls": ["https://blog.cloudflare.com"],
  "maxPages": 50,
  "excludePatterns": ["**/tag/**", "**/author/**"],
  "lastmodAfter": "90 days",
  "chunkSize": 500,
  "chunkOverlap": 50,
  "createLlmsTxt": true
}

lastmodAfter keeps only sitemap URLs with a <lastmod> on or after the date, so sites whose sitemap has no <lastmod> return nothing with it set; use Incremental mode for those.

Only want the URL list? Set "sitemapOnly": true.

Output example

One record per page:

{
  "type": "page",
  "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/name-your-actor",
  "title": "Name your Actor | Academy | Apify Documentation",
  "description": "Learn Apify's standards for naming Actors…",
  "language": "en",
  "lastmod": "2026-10-10T00:00:00.000Z",
  "markdown": "# Name your Actor\n\nApify's standards for Actor naming…",
  "wordCount": 1326,
  "charCount": 8840,
  "tokenEstimate": 2210,
  "headings": [{ "level": 1, "text": "Name your Actor" }, { "level": 2, "text": "Actor name vs. SEO name" }],
  "linksCount": 6,
  "internalLinks": 5,
  "externalLinks": 1,
  "chunkCount": 5,
  "chunks": [{ "index": 0, "heading": "Name your Actor", "text": "# Name your Actor\n\n…", "charCount": 1890, "tokenEstimate": 473 }],
  "contentHash": "bef532f82cd982d0850ced171078ab576ace40cc5aa05ef27f48b8cafa0b16ad",
  "source": "sitemap",
  "statusCode": 200,
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "scrapedAt": "2026-10-18T05:30:00.000Z"
}

Sitemap-only records: { "type": "url", "url", "lastmod", "changefreq", "priority", "sitemapUrl" }.

Error records: { "type": "error", "url", "error": "FETCH_FAILED", "errorMessage": "Could not load page: HTTP 404." }.

Key-value store: llms.txt, llms-full.txt, RUN_SUMMARY (sitemaps found, how each site was discovered, URL counts, duplicates skipped, errors, robots.txt skips) and, in incremental mode, CHANGES (shape shown with one edited and one removed post):

{ "stateKey": "state-656b5c5cfb3311cc", "new": [], "changed": ["https://htmx.org/posts/2020-11-16-htmx-0-4-0-is-released/"], "unchanged": ["…35 URLs"], "removed": ["https://htmx.org/posts/2099-01-01-gone/"] }

How much does it cost? (about $0.50 per 1,000 pages)

Pay per result, no subscription:

Event	Price
Actor start	$0.00005
Each dataset record (page, URL in sitemap-only mode, or error record)	$0.0005

1,000 pages of Markdown = $0.50. Chunks are stored inside the page record, so chunking costs nothing extra, and the llms.txt files are free. Platform usage is included. Turn off Save error records if you don't want failed pages to count. In incremental mode, unchanged pages aren't in the dataset, so a re-run of a site where nothing changed costs only the start fee. Skipped duplicates aren't charged either.

How it compares

Generic website crawlers follow links and can render JavaScript, which is slower and pricier; Website Content Crawler, for example, is billed by compute and lists about $0.20 per 1,000 pages with raw HTTP and $0.50–$5 with a browser. This Actor reads the sitemap the site already publishes, so it gets exactly the pages the owner wants indexed, at a flat $0.50 per 1,000 pages, and adds what a RAG sync needs out of the box: per-page changeStatus and a removed-URL list between runs, contentHash, token chunks, YAML front matter and llms.txt. If you need JavaScript rendering, proxies, logins or file downloads (PDF, DOCX), use a browser-based crawler instead.

Tips

• Docs sites (Docusaurus, MkDocs, GitBook, ReadTheDocs, Mintlify, Nextra): pass the docs root, e.g. https://docs.example.com.
• WordPress: /wp-sitemap.xml and Yoast/RankMath indexes are found automatically; exclude **/tag/** and **/author/**.
• Weekly refresh: schedule the Actor with Incremental mode on, upsert pages by url, and delete the removed URLs from the CHANGES record.
• Messy pages: set Main content CSS selector (e.g. article) or add Extra CSS selectors to remove.
• JavaScript-only sites (content rendered client-side with no server HTML) return little text; this Actor does not run a browser. Those pages come back as thin_page, so you can spot them (tested on excalidraw.com and app.diagrams.net).

Use with AI agents and the API

Run it from the Apify API, the Apify MCP server, LangChain's ApifyWrapper, LlamaIndex, Make, Zapier or n8n. Read results from the default dataset (markdown and chunks fields) or fetch llms-full.txt from the key-value store.

Limits and responsibility

The Actor only fetches publicly available pages listed in a site's sitemap or linked from its pages, and respects robots.txt by default. Pages behind logins or paywalls are not accessed. Make sure your use of the content complies with the website's terms and copyright law.

Related tools

• Free competitor price monitor for n8n: a free n8n template that checks a list of competitor product pages every morning and posts to Slack when a price or stock level changes. Use this Actor's URL-only mode (Sitemap only, with an include pattern like **/products/**) to pull every product URL from a competitor's sitemap, then paste the ones you care about into the template's Google Sheets watchlist. The template is a separate product: this Actor doesn't track prices itself.

Support

Found a site that doesn't convert well, a bug, or need a field? Open an issue in the Issues tab with the URL and I'll usually reply within a day. For programmatic use, see the API tab.