# Website to Markdown & Sitemap to Markdown: RAG, llms.txt (`chimey_forearm/sitemap-to-markdown`) Actor

Turn any website into clean Markdown for RAG using its sitemap. Incremental mode returns only new and changed pages and lists removed URLs. Heading-aware token chunks and llms.txt in the same run at no extra charge. Link-crawl fallback, contentHash. No browser. $0.50 per 1,000 pages.

- **URL**: https://apify.com/chimey_forearm/sitemap-to-markdown.md
- **Developed by:** [Mohamed Bahaa](https://apify.com/chimey_forearm) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Sitemap to Markdown do?

**Sitemap to Markdown** converter that turns any **website into clean, LLM-ready Markdown** using its sitemap (website to Markdown). Give it a domain like `https://docs.example.com` and it finds the sitemap on its own (robots.txt, `/sitemap.xml`, sitemap indexes, `.xml.gz`), filters the URLs you care about, downloads each page, strips the navigation, footers, cookie banners and scripts, and returns the main content as Markdown, with title, description, headings, word count, token estimate and optional **RAG chunks**. It can also build an **`llms.txt` and `llms-full.txt`** for the site, or act as a plain **sitemap scraper** that lists URLs with `lastmod`.

#### What makes it different

- **Incremental sync that also tells you what was removed.** With `incremental` on, unchanged pages are not fetched or charged, new and changed pages come back with `changeStatus`, and a `CHANGES` record lists new, changed, unchanged and **removed** URLs, so you can delete stale vectors too. Test: once all 36 htmx.org posts were saved, the next run returned 0 pages (36 unchanged).
- **RAG chunks and llms.txt in the same run, at no extra charge.** Heading-aware chunks with token estimates sit inside each page record, and `llms.txt` / `llms-full.txt` are written to the key-value store.
- **Flat $0.50 per 1,000 pages.** Plain HTTP at 256 MB, no browser and no proxy costs on top.

Real output from a test run (`https://htmx.org/posts/`, incremental, 300-token chunks, Oct 18 2026; text shortened):

```json
{
  "url": "https://htmx.org/posts/2023-11-03-htmx-1-9-7-is-released/",
  "changeStatus": "new",
  "contentHash": "568bcacf33727b1eb3a36b522e2eb99d2cd8cfcbb034de323c72ec04975327d0",
  "title": "</> htmx ~ htmx 1.9.7 has been released!",
  "lastmod": "2023-11-03",
  "wordCount": 145,
  "tokenEstimate": 246,
  "source": "sitemap",
  "chunkCount": 1,
  "chunks": [
    {
      "index": 0,
      "heading": "htmx 1.9.7 has been released!",
      "text": "# htmx 1.9.7 has been released!\n\n## htmx 1.9.7 Release\n\nI’m happy to announce the [1.9.7 r …",
      "charCount": 982,
      "tokenEstimate": 246
    }
  ]
}
```

No sitemap? It **crawls the site's links** instead. Re-running on a schedule? **Incremental mode** only fetches and returns pages that are new or changed since the last run, and lists removed pages, so you only pay for what changed. Every page gets a **`contentHash`**, and exact or canonical duplicates are skipped.

No browser, no proxies, no login: plain HTTP, so it's fast and cheap. **$0.50 per 1,000 pages.**

Works on any website with a sitemap. Every example below was test-run and works as is: paste the JSON into the **Input** tab (switch to JSON view) and click **Start**.

### Example use cases

#### 1. Turn a docs site into Markdown chunks for a RAG chatbot

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20,
  "chunkSize": 500
}
```

#### 2. Keep a vector store in sync: only new and changed pages

Run it on a schedule with the same input. The first run saves a baseline; later runs skip pages whose sitemap `lastmod` hasn't changed (not fetched, not charged), drop pages whose content hash is the same, and list removed URLs in the `CHANGES` record.

```json
{
  "startUrls": [
    "https://htmx.org/posts/"
  ],
  "maxPages": 0,
  "incremental": true
}
```

In our test the first run returned 36 pages (`changeStatus: "new"`); the second run returned 0 pages and reported 36 unchanged.

#### 3. A site with no sitemap

```json
{
  "startUrls": [
    "https://www.paulgraham.com"
  ],
  "maxPages": 25
}
```

The site has no sitemap, so the Actor crawls same-site links from the start URL (2 clicks deep by default). Each page says `"source": "linkCrawl"` and its `crawlDepth`. `/index.html` is skipped as a duplicate of `/`.

### What data can Sitemap to Markdown extract?

| Field | What it is |
|---|---|
| `url / title` | Page URL and title |
| `markdown` | Main page content as clean Markdown |
| `lastmod` | Last-modified date from the sitemap |
| `contentHash` | SHA-256 of the page's Markdown, for dedupe and change detection |
| `changeStatus` | `new` or `changed` (incremental mode only) |
| `source / crawlDepth` | Found in the `sitemap` or by `linkCrawl`, and how many clicks from the start URL |
| `canonicalUrl` | The page's `rel=canonical` URL |
| `wordCount / tokenEstimate` | Size of the page |
| `chunks` | RAG-ready chunks, when chunking is on |
| `language` | Page language |

#### Sample output

Real rows from a run with the default input (text shortened):

| url | title | wordCount | tokenEstimate | contentHash | source |
|---|---|---|---|---|---|
| https://docs.apify.com/academy | Apify Academy | Academy | Apify Documentation | 215 | 460 | 4807eb7e09… | sitemap |
| https://docs.apify.com/academy/actor-marketing-playbook/acto… | Importance of Actor URL | Academy | Apify Documentation | 1152 | 1672 | 848bcee4d3… | sitemap |
| https://docs.apify.com/academy/actor-marketing-playbook/acto… | Actor description & SEO description | Academy | Apify Documentation | 1237 | 2008 | bef532f82c… | sitemap |

You can download the dataset as JSON, CSV, Excel, XML or HTML.

#### Front matter and thin-page flags

Each page's `markdown` starts with a YAML front matter block (turn off with **Add YAML front matter**):

```yaml
---
title: "Actors and emojis | Academy | Apify Documentation"
url: "https://docs.apify.com/academy/..."
canonical_url: "https://docs.apify.com/academy/..."
lastmod: "2026-10-01"
language: "en"
word_count: 554
thin_content: false
scraped_at: "2026-10-18T07:40:00.000Z"
---
```

Use `lastmod` to re-pull only changed pages and `canonical_url` to skip duplicate paths. Pages with less main content than **Thin page threshold** (default 50 words) come back as `type: "thin_page"` with `thinContent: true` and are left out of llms.txt, so they don't end up in your embeddings as noise. Turn off **Save thin pages** to skip them entirely, and you won't be charged for them.

#### Incremental mode, link crawl and duplicates

- **Incremental mode** (`incremental: true`): the Actor remembers each page's sitemap `lastmod` and content hash in the named key-value store `sitemap-to-markdown-state` (one record per set of start URLs; set **Incremental state name** to keep separate histories). On the next run, pages with the same `lastmod` are skipped without being fetched, and fetched pages with the same hash are left out of the dataset (turn on **Also output unchanged pages** to keep them). You're charged only for pages in the dataset. The `CHANGES` record in the run's key-value store lists `new`, `changed`, `unchanged` and `removed` URLs, so you can upsert and delete vectors. Removed pages are detected when the whole sitemap was read (set **Max pages** to 0 or above the site's size, and leave **Changed after** empty). In this mode **Max pages** counts pages that need fetching, so unchanged pages don't use it up, and `llms.txt` covers only the pages returned in that run.
- **Link crawl** (`linkCrawl`): `fallback` (default) crawls same-site links from the start URL when there is no sitemap, or the sitemap has nothing under your start URL; `always` adds linked pages to the sitemap's pages; `never` turns it off. Depth is set by **Max link-crawl depth** (default 2). Links stay on the same site and under the start path, and follow your include/exclude patterns and robots.txt. **Changed after** only filters sitemap URLs, since crawled pages have no `lastmod`.
- **Duplicates** (`skipDuplicates`, on by default): a page whose Markdown is identical to one already saved in the run, or that shares a canonical URL with a saved page and has near-identical length, is skipped and not charged.
- **Blocked sites**: a 401/403/429 or a Cloudflare challenge comes back as a `BLOCKED` error record. This Actor uses plain HTTP without proxies or a browser, so it can't get past bot walls.

### How to scrape a website to Markdown

1. Open the **Input** tab and enter a site or sitemap URL, for example `https://docs.apify.com/academy`.
2. Set **Max pages**, optional URL include/exclude patterns, and a chunk size if you want RAG chunks.
3. Click **Start**.
4. Download the pages as JSON, CSV or Markdown, or grab `llms.txt` and `llms-full.txt` from the key-value store.

You can also run it on a schedule, call it from the **API** tab, or connect it to Make, Zapier, n8n, Google Sheets or an AI agent through the Apify MCP server.

### Who it's for

- **RAG pipelines and vector databases**: feed docs, help centers, blogs and knowledge bases into Pinecone, Weaviate, Qdrant, pgvector, Chroma or LlamaIndex/LangChain loaders.
- **AI agents and MCP clients**: give an agent a website's full content as Markdown in one call.
- **Custom GPTs, Claude Projects and NotebookLM**: download `llms-full.txt` and upload one file.
- **llms.txt generation**: publish an `llms.txt` index for your own site.
- **Content audits and SEO**: list every URL in a sitemap with `lastmod` (sitemap-only mode), word counts and headings.
- **Incremental refreshes**: schedule it with **Incremental mode** and only new or changed pages come back (and get charged).

### Features

- **Sitemap auto-discovery**: robots.txt `Sitemap:` lines, `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`; nested sitemap indexes; gzip; plain-text sitemaps.
- **Link-crawl fallback** for sites without a usable sitemap (same-site, depth-limited).
- **Incremental mode**: skip pages unchanged since the last run by sitemap `lastmod` or content hash; `changeStatus` per page and a `CHANGES` record with removed URLs.
- **Dedupe**: `contentHash` on every page; identical and same-canonical duplicates skipped.
- **Section scoping**: `https://example.com/blog` keeps only pages under `/blog`.
- **Filters**: include/exclude URL patterns (glob like `**/docs/**` or `/regex/`), `lastmod` after a date or "30 days", newest-first ordering, max pages per site.
- **Main-content extraction**: semantic `<main>`/`<article>` detection with boilerplate removal, Mozilla Readability fallback, or your own CSS selector.
- **Clean Markdown**: ATX headings, fenced code blocks with language, GitHub tables, absolute links, no images by default (fewer tokens), heading permalink icons removed.
- **RAG chunking**: split by headings and paragraphs to a token or character size, with overlap and the heading path for each chunk.
- **YAML front matter on every page**: `title`, `url`, `canonical_url`, `lastmod` (from the sitemap), `language`, `word_count`, `thin_content` and `scraped_at`, so your pipeline can re-pull only changed pages and skip duplicate paths. Also available as the `frontMatter` field. Turn off with **Add YAML front matter**.
- **Thin-page flag**: pages with less main content than **Thin page threshold** (default 50 words) come back as `type: "thin_page"` with `thinContent: true` and are left out of llms.txt, so near-empty pages don't end up embedded as noise. Turn off **Save thin pages** to skip them entirely and not pay for them.
- **llms.txt + llms-full.txt** saved to the key-value store.
- **Polite**: respects robots.txt `Disallow` and `Crawl-delay`, concurrency limit, retries with backoff on 429/5xx.
- **Clear error records**: `NO_SITEMAP_FOUND`, `BLOCKED`, `FETCH_FAILED`, `NOT_HTML`, `INVALID_URL`.
- **Any language**: tested on Japanese and German sites; Chinese, Japanese and Korean text is counted per character for word counts, so CJK pages aren't wrongly flagged as thin.
- **Spending-limit aware**: stops cleanly at your run's max charge.

### Input example

```json
{
  "startUrls": ["https://blog.cloudflare.com"],
  "maxPages": 50,
  "excludePatterns": ["**/tag/**", "**/author/**"],
  "lastmodAfter": "90 days",
  "chunkSize": 500,
  "chunkOverlap": 50,
  "createLlmsTxt": true
}
```

`lastmodAfter` keeps only sitemap URLs with a `<lastmod>` on or after the date, so sites whose sitemap has no `<lastmod>` return nothing with it set; use **Incremental mode** for those.

Only want the URL list? Set `"sitemapOnly": true`.

### Output example

One record per page:

```json
{
  "type": "page",
  "url": "https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/name-your-actor",
  "title": "Name your Actor | Academy | Apify Documentation",
  "description": "Learn Apify's standards for naming Actors…",
  "language": "en",
  "lastmod": "2026-10-10T00:00:00.000Z",
  "markdown": "# Name your Actor\n\nApify's standards for Actor naming…",
  "wordCount": 1326,
  "charCount": 8840,
  "tokenEstimate": 2210,
  "headings": [{ "level": 1, "text": "Name your Actor" }, { "level": 2, "text": "Actor name vs. SEO name" }],
  "linksCount": 6,
  "internalLinks": 5,
  "externalLinks": 1,
  "chunkCount": 5,
  "chunks": [{ "index": 0, "heading": "Name your Actor", "text": "# Name your Actor\n\n…", "charCount": 1890, "tokenEstimate": 473 }],
  "contentHash": "bef532f82cd982d0850ced171078ab576ace40cc5aa05ef27f48b8cafa0b16ad",
  "source": "sitemap",
  "statusCode": 200,
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "scrapedAt": "2026-10-18T05:30:00.000Z"
}
```

Sitemap-only records: `{ "type": "url", "url", "lastmod", "changefreq", "priority", "sitemapUrl" }`.

Error records: `{ "type": "error", "url", "error": "FETCH_FAILED", "errorMessage": "Could not load page: HTTP 404." }`.

**Key-value store:** `llms.txt`, `llms-full.txt`, `RUN_SUMMARY` (sitemaps found, how each site was discovered, URL counts, duplicates skipped, errors, robots.txt skips) and, in incremental mode, `CHANGES` (shape shown with one edited and one removed post):

```json
{ "stateKey": "state-656b5c5cfb3311cc", "new": [], "changed": ["https://htmx.org/posts/2020-11-16-htmx-0-4-0-is-released/"], "unchanged": ["…35 URLs"], "removed": ["https://htmx.org/posts/2099-01-01-gone/"] }
```

### How much does it cost? (about $0.50 per 1,000 pages)

Pay per result, no subscription:

| Event | Price |
|---|---|
| Actor start | $0.00005 |
| Each dataset record (page, URL in sitemap-only mode, or error record) | $0.0005 |

1,000 pages of Markdown = **$0.50**. Chunks are stored inside the page record, so chunking costs nothing extra, and the llms.txt files are free. Platform usage is included. Turn off **Save error records** if you don't want failed pages to count. In incremental mode, unchanged pages aren't in the dataset, so a re-run of a site where nothing changed costs only the start fee. Skipped duplicates aren't charged either.

### How it compares

Generic website crawlers follow links and can render JavaScript, which is slower and pricier; Website Content Crawler, for example, is billed by compute and lists about $0.20 per 1,000 pages with raw HTTP and $0.50–$5 with a browser. This Actor reads the sitemap the site already publishes, so it gets exactly the pages the owner wants indexed, at a flat $0.50 per 1,000 pages, and adds what a RAG sync needs out of the box: per-page `changeStatus` and a removed-URL list between runs, `contentHash`, token chunks, YAML front matter and llms.txt. If you need JavaScript rendering, proxies, logins or file downloads (PDF, DOCX), use a browser-based crawler instead.

### Tips

- **Docs sites** (Docusaurus, MkDocs, GitBook, ReadTheDocs, Mintlify, Nextra): pass the docs root, e.g. `https://docs.example.com`.
- **WordPress**: `/wp-sitemap.xml` and Yoast/RankMath indexes are found automatically; exclude `**/tag/**` and `**/author/**`.
- **Weekly refresh**: schedule the Actor with **Incremental mode** on, upsert pages by `url`, and delete the `removed` URLs from the `CHANGES` record.
- **Messy pages**: set **Main content CSS selector** (e.g. `article`) or add **Extra CSS selectors to remove**.
- **JavaScript-only sites** (content rendered client-side with no server HTML) return little text; this Actor does not run a browser. Those pages come back as `thin_page`, so you can spot them (tested on excalidraw.com and app.diagrams.net).

### Use with AI agents and the API

Run it from the Apify API, the Apify MCP server, LangChain's `ApifyWrapper`, LlamaIndex, Make, Zapier or n8n. Read results from the default dataset (`markdown` and `chunks` fields) or fetch `llms-full.txt` from the key-value store.

### Limits and responsibility

The Actor only fetches publicly available pages listed in a site's sitemap or linked from its pages, and respects robots.txt by default. Pages behind logins or paywalls are not accessed. Make sure your use of the content complies with the website's terms and copyright law.

### Related tools

- **[Free competitor price monitor for n8n](https://mohabaha0.gumroad.com/l/price-monitor-free)**: a free n8n template that checks a list of competitor product pages every morning and posts to Slack when a price or stock level changes. Use this Actor's URL-only mode (**Sitemap only**, with an include pattern like `**/products/**`) to pull every product URL from a competitor's sitemap, then paste the ones you care about into the template's Google Sheets watchlist. The template is a separate product: this Actor doesn't track prices itself.

### Support

Found a site that doesn't convert well, a bug, or need a field? Open an issue in the **Issues** tab with the URL and I'll usually reply within a day. For programmatic use, see the **API** tab.

# Actor input Schema

## `startUrls` (type: `array`):

One per line. A website (https://docs.example.com) auto-discovers its sitemap from robots.txt, /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and .xml.gz, following nested sitemap indexes. A section URL (https://example.com/blog) keeps only pages under that path. A sitemap URL (…/sitemap.xml, .xml.gz or a plain-text list) is used directly.

## `maxPages` (type: `integer`):

Stop after this many pages (or URLs in sitemap-only mode) for each start URL, after filters. 0 = no limit (the run's spending limit still applies).

## `sitemapOnly` (type: `boolean`):

Just list every URL from the sitemaps with lastmod, changefreq and priority. Fast and cheap: use it to audit a site or plan a crawl, then run again with filters.

## `includePatterns` (type: `array`):

Keep only URLs matching ANY of these. Glob ("/docs/**", "**/blog/\*") or regex in slashes ("//20(24|25)//"). Overrides the automatic section-path filter.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these, e.g. "**/tag/**", "**/author/**", "/.pdf$/".

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap <lastmod> is on/after this date. Absolute (2025-01-31) or relative ("30 days", "2 weeks"). URLs without lastmod are dropped when this is set. Great for incremental RAG refreshes.

## `incremental` (type: `boolean`):

Remember every page between runs (in the named key-value store "sitemap-to-markdown-state"). Pages whose sitemap <lastmod> hasn't changed are skipped without being fetched; fetched pages whose content hash is the same as last time are left out of the dataset. You're only charged for new and changed pages. Each page gets changeStatus (new/changed), and the CHANGES record lists new, changed, unchanged and removed URLs, so you can update or delete vectors. The first run saves a baseline.

## `incrementalStateKey` (type: `string`):

Optional. Runs with the same start URLs share state automatically. Set a name to keep separate histories for the same site (letters, digits, - \_ . only).

## `outputUnchanged` (type: `boolean`):

In incremental mode, also save pages that were fetched but whose content didn't change (changeStatus "unchanged"; these count as results). Pages skipped by lastmod are never fetched or saved.

## `sortBy` (type: `string`):

Sitemap order, or newest lastmod first (reads all sitemaps first, then takes the newest pages).

## `linkCrawl` (type: `string`):

What to do when a website has no sitemap, or its sitemap lists nothing under your start URL. "fallback" (default): crawl same-site links from the start URL instead, up to the depth below. "always": use the sitemap AND follow links, to catch pages the sitemap misses. "never": sitemap only. Not used for direct sitemap URLs or in sitemap-only mode.

## `maxCrawlDepth` (type: `integer`):

How many clicks away from the start URL the link crawl goes (0 = start page only). Pages still stop at Max pages.

## `chunkSize` (type: `integer`):

Split each page's Markdown into RAG-ready chunks of about this size, on heading and paragraph boundaries (code blocks kept whole when they fit). Each chunk carries its heading path. Chunks are an array inside the page record, so they don't cost extra.

## `chunkUnit` (type: `string`):

Tokens are estimated as characters ÷ 4 (close to OpenAI/Anthropic tokenizers for English).

## `chunkOverlap` (type: `integer`):

Repeat this much of the end of the previous chunk at the start of the next (same unit as chunk size).

## `createLlmsTxt` (type: `boolean`):

Also save an llms.txt index (title, link, description per page) and llms-full.txt (every page's Markdown in one file) to the run's key-value store.

## `contentSelector` (type: `string`):

Override automatic main-content detection, e.g. "article.post" or ".markdown". Leave empty for auto (semantic <main>/<article> first, Mozilla Readability fallback).

## `removeSelectors` (type: `array`):

Elements to strip before conversion, e.g. ".newsletter-signup", "#related". Nav, header, footer, sidebars, scripts and cookie banners are removed automatically.

## `includeLinksInMarkdown` (type: `boolean`):

Keep [text](url) links (absolute URLs). Turn off for plain text-like Markdown with fewer tokens.

## `includeImages` (type: `boolean`):

Keep ![alt](src) image references. Off by default to save tokens.

## `addFrontMatter` (type: `boolean`):

Start each page's Markdown with a YAML front matter block: title, url, canonical_url, lastmod (from the sitemap), language, word_count, thin_content, scraped_at. The same values are also in the frontMatter field.

## `minWords` (type: `integer`):

Pages whose main content has fewer words than this are flagged as thin (type "thin_page", thinContent: true) and left out of llms.txt and llms-full.txt. Set 0 to turn off.

## `saveThinPages` (type: `boolean`):

Keep flagged thin pages in the dataset (they count as results). Turn off to skip them entirely, so you aren't charged for them.

## `skipDuplicates` (type: `boolean`):

Leave out pages whose Markdown is identical to a page already saved in this run (same contentHash), or that share a canonical URL with a saved page and have near-identical length. Skipped duplicates aren't charged.

## `respectRobotsTxt` (type: `boolean`):

Skip pages disallowed for all user agents (User-agent: \*) and honor Crawl-delay (up to 10 s).

## `sameDomainOnly` (type: `boolean`):

Ignore sitemap URLs on other domains (www. is treated as the same domain).

## `restrictToStartPath` (type: `boolean`):

For a start URL with a path (https://example.com/docs), keep only pages under /docs. Ignored when include patterns are set.

## `maxConcurrency` (type: `integer`):

Parallel page downloads. Keep it low to be polite to small sites.

## `saveErrorRecords` (type: `boolean`):

Add a record for every input or page that failed (NO_SITEMAP_FOUND, FETCH_FAILED, NOT_HTML, INVALID_URL). Error records count as results.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20,
  "sitemapOnly": false,
  "incremental": false,
  "outputUnchanged": false,
  "sortBy": "sitemap",
  "linkCrawl": "fallback",
  "maxCrawlDepth": 2,
  "chunkSize": 0,
  "chunkUnit": "tokens",
  "chunkOverlap": 0,
  "createLlmsTxt": true,
  "includeLinksInMarkdown": true,
  "includeImages": false,
  "addFrontMatter": true,
  "minWords": 50,
  "saveThinPages": true,
  "skipDuplicates": true,
  "respectRobotsTxt": true,
  "sameDomainOnly": true,
  "restrictToStartPath": true,
  "maxConcurrency": 5,
  "saveErrorRecords": true
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `llmsFullTxt` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `changes` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy"
    ],
    "maxPages": 20,
    "chunkSize": 0
};

// Run the Actor and wait for it to finish
const run = await client.actor("chimey_forearm/sitemap-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy"],
    "maxPages": 20,
    "chunkSize": 0,
}

# Run the Actor and wait for it to finish
run = client.actor("chimey_forearm/sitemap-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy"
  ],
  "maxPages": 20,
  "chunkSize": 0
}' |
apify call chimey_forearm/sitemap-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,chimey_forearm/sitemap-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0JLwabm18ck8rcUAO/builds/z1oDoHxB45m88qDdi/openapi.json