# Sitemap to Markdown Crawler — Website to Markdown for LLM & RAG (`leadsbrary/sitemap-to-markdown-crawler`) Actor

Website to markdown crawler: give a domain, it finds the sitemap, fetches pages over plain HTTP and returns clean Markdown, headings, links, lastmod and content hash. Incremental re-crawls. Export JSON/CSV/Excel.

- **URL**: https://apify.com/leadsbrary/sitemap-to-markdown-crawler.md
- **Developed by:** [Alexandre Manguis](https://apify.com/leadsbrary) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap to Markdown Crawler — Website to Markdown for LLM & RAG

### What does Sitemap to Markdown Crawler do?

This **website to markdown crawler** takes a domain, finds its `robots.txt` and `sitemap.xml` files, downloads each page over plain HTTP (no browser) and returns clean Markdown with title, description, headings, link counts, word count, `lastmod` and a content hash. Export as JSON, CSV, Excel, XML or HTML from the Apify dataset, or call it through the API.

It is built for feeding documentation sites, blogs and help centres into LLM / RAG pipelines, and for re-crawling only what changed.

### Who is it for? Use cases

- **RAG and vector databases**: index a documentation site from its sitemap.
- **Weekly knowledge-base refresh**: turn on incremental mode, so unchanged pages are neither output nor billed.
- **Support chatbots**: convert a help centre to Markdown.
- **Content audits**: word counts, headings, meta descriptions and internal/external link counts per page.
- **Blog export**: Markdown for migration or LLM fine-tuning datasets.

### Features

- **Sitemap discovery**: `robots.txt` → `sitemap.xml`, sitemap indexes and `.xml.gz`. Pass a sitemap URL directly if you prefer.
- **Link-following fallback** (depth 0–10) when no sitemap exists; or choose `sitemap_only` / `links_only`.
- **HTTP only**: no headless browser, so runs are cheap and fast. JavaScript-rendered pages with almost no server-side text (< 20 words) are skipped, not billed.
- **Incremental runs**: `skipUnchanged` stores a content hash per URL; `changedSince` filters on sitemap `lastmod`.
- **Main-content extraction** with `main`/`article` heuristics, or your own `contentSelector`; `removeSelectors` strips cookie banners and the like.
- **URL include/exclude globs**, optional cleaned HTML, `robots.txt` respected by default.
- Optional Apify Proxy if a site blocks direct requests.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` (required) | array | crawlee.dev | Domains, homepages, sitemap URLs or seed pages (max 100). |
| `maxResults` | integer | 50 | Max pages delivered and billed (1–100000). |
| `discoveryMode` | string | `sitemap_then_links` | `sitemap_then_links`, `sitemap_only`, `links_only`. |
| `includeUrlPatterns` / `excludeUrlPatterns` | string list | empty | Glob patterns, e.g. `https://crawlee.dev/js/docs/**`, `**/*.pdf`. |
| `maxCrawlDepth` | integer | 3 | Link depth when following links (0–10). |
| `changedSince` | string | – | ISO date; sitemap URLs with an older `lastmod` are skipped (URLs without `lastmod` are kept). |
| `skipUnchanged` | boolean | false | Skip pages whose content hash is unchanged since the last run with the same `stateKey`. |
| `stateKey` | string | first host | Name of the incremental state. |
| `contentSelector` | string | – | CSS selector of the main content. |
| `removeSelectors` | string list | empty | CSS selectors removed before conversion. |
| `includeHtml` | boolean | false | Also output cleaned HTML. |
| `respectRobotsTxt` | boolean | true | Skip URLs disallowed by robots.txt. |
| `maxConcurrency` | integer | 10 | Parallel requests (1–50). |
| `proxyConfiguration` | object | none | Apify Proxy settings. |

Example input:

```json
{
  "startUrls": [{ "url": "https://crawlee.dev" }],
  "maxResults": 50,
  "discoveryMode": "sitemap_then_links",
  "skipUnchanged": true
}
```

### Output

Real item from a production run (Markdown shortened):

```json
{
  "url": "https://crawlee.dev/blog/building-gradcracker-scraper-with-crawlee",
  "requestedUrl": "https://crawlee.dev/blog/building-gradcracker-scraper-with-crawlee",
  "domain": "crawlee.dev",
  "title": "Building a Gradcracker scraper with Crawlee: anti-bot failures and redirect bugs | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "A real-world breakdown of building a Gradcracker scraper with Crawlee, handling bot detection, unstable redirects, and production trade-offs.",
  "language": "en",
  "markdown": "I spent three days debugging a scraper that I really expected to have knocked out in an afternoon. ...",
  "html": null,
  "headings": [{ "level": 2, "text": "What JobOps does" }, { "level": 2, "text": "Conclusion" }],
  "wordCount": 1738,
  "links": { "internal": 5, "external": 14 },
  "canonicalUrl": "https://crawlee.dev/blog/building-gradcracker-scraper-with-crawlee",
  "lastmod": null,
  "lastModifiedHeader": "2026-10-02T13:44:42.000Z",
  "contentHash": "2b60f59248d46f98f513e382968ead77f29f5812060f8e9206ac94751f88aa70",
  "changeStatus": "unknown",
  "discoveredFrom": "sitemap",
  "httpStatus": 200,
  "crawledAt": "2026-10-04T08:22:59.686Z"
}
```

#### Output fields & fill rates

Measured on a 40-page production run (crawlee.dev):

| Field | Fill rate | Notes |
|---|---|---|
| `url`, `requestedUrl`, `domain`, `canonicalUrl` | 100% | |
| `title`, `markdown`, `headings`, `links`, `wordCount` | 100% | |
| `language`, `httpStatus`, `contentHash`, `crawledAt`, `discoveredFrom` | 100% | |
| `lastModifiedHeader` | 100% | From the HTTP `Last-Modified` header, on this site. |
| `description` | 92.5% | Meta description, empty when the page has none. |
| `lastmod` | 17.5% | Only when the sitemap provides it. |
| `changeStatus` | 100% | `new` / `changed` when `skipUnchanged` is on (`new` seen in proof runs), otherwise `unknown`. |
| `html` | 0% | Only filled with `includeHtml: true`. |

Fill rates depend on the target site.

#### Examples of each mode (real proof runs on crawlee.dev)

| Mode | Input | Result |
|---|---|---|
| Sitemap (default) | `{"startUrls":[{"url":"https://crawlee.dev"}],"maxResults":10}` | 10 pages, `discoveredFrom: "sitemap"` |
| Incremental, first run | same input + `"skipUnchanged": true, "stateKey": "qa-incremental"` | 10 pages, all `changeStatus: "new"` (run `EbEsitDuybxtzu5TY`) |
| Incremental, second run | identical input, same `stateKey` | 10 different pages, `changeStatus: "new"`. The pages delivered by the first run were fetched again (20 requests in total) but not output or billed, so a repeated run only returns pages not seen before (run `aR6iZwob6L0O9C1X8`). |
| Link following | `{"startUrls":[{"url":"https://crawlee.dev/js/docs/quick-start"}],"discoveryMode":"links_only","maxCrawlDepth":1,"maxResults":10,"includeUrlPatterns":["https://crawlee.dev/js/docs/**"]}` | 10 pages, first one `discoveredFrom: "start_url"` (run `B4i0MAl3Wt6b5BwgE`) |
| Unreachable host | `INPUT.edge-unreachable.json` | 0 items, no error row, run `SUCCEEDED` (run `Iht73mb4UQbj6JKb2`) |

`changeStatus` is `unknown` when `skipUnchanged` is off.

### How to use via API

**curl**

```bash
curl -X POST "https://api.apify.com/v2/acts/leadsbrary~sitemap-to-markdown-crawler/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://crawlee.dev"}],"maxResults":50}'
```

**JavaScript**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('leadsbrary/sitemap-to-markdown-crawler').call({
  startUrls: [{ url: 'https://crawlee.dev' }],
  maxResults: 50,
  skipUnchanged: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].markdown);
```

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("leadsbrary/sitemap-to-markdown-crawler").call(run_input={
    "startUrls": [{"url": "https://crawlee.dev"}],
    "maxResults": 50,
    "skipUnchanged": True,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items[0]["markdown"])
```

### Pricing

Pay per result: one result = one page delivered. Price per 1,000 pages depends on your Apify plan:

| Plan | Per 1,000 results |
|---|---|
| Free | $1.20 |
| Bronze | $1.04 |
| Silver | $0.92 |
| Gold and above | $0.80 |

Skipped pages (unchanged, near-empty, blocked, disallowed) are not billed. `maxResults` caps both output and cost. Apify platform usage is separate; the production runs used about $0.0005 of platform usage for 10–40 pages.

### Troubleshooting

- **0 results**: the site may have no reachable sitemap and link-following may be off (`sitemap_only`), or the site was unreachable. A test run against an unreachable host finished with 0 items and no error rows. Check the run log.
- **Pages missing or near-empty**: the site renders content with JavaScript. This Actor does not run a browser and skips pages with fewer than 20 words.
- **Blocked (403/429)**: lower `maxConcurrency` and enable Apify Proxy in `proxyConfiguration`.
- **Wrong content extracted**: set `contentSelector` and `removeSelectors`.
- **URLs skipped**: check `respectRobotsTxt` and your include/exclude patterns.
- **"startUrls is required"**: provide at least one URL.

### FAQ

**Does it run a browser?** No, plain HTTP only, which keeps it cheap and fast.

**How do I only get changed pages?** Set `skipUnchanged: true` and reuse the same `stateKey`. Use `changedSince` to also filter on sitemap `lastmod`.

**Can I crawl several sites in one run?** Yes, up to 100 start URLs; `maxResults` is shared across them.

**Is it legal to crawl a site?** It only reads public pages and respects `robots.txt` by default. Check the target's terms of service and copyright rules for your use of the content.

### Keywords

website to markdown crawler, sitemap to markdown, sitemap crawler, website to markdown, html to markdown, llm docs scraper, rag website crawler, documentation scraper, website content crawler, markdown scraper

### Hashtags

\#markdown #sitemap #crawler #llm #rag #webscraping #documentation #ai #apify

# Changelog

This Actor's version history is a separate document: https://apify.com/leadsbrary/sitemap-to-markdown-crawler/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Websites or pages to convert. A homepage or bare domain triggers robots.txt/sitemap discovery; a sitemap URL is used directly; other URLs are crawl seeds.

## `maxResults` (type: `integer`):

Maximum number of pages delivered (and billed) in this run.

## `discoveryMode` (type: `string`):

Sitemap first with link-following fallback, sitemap only, or link-following only.

## `includeUrlPatterns` (type: `array`):

Glob patterns (e.g. https://crawlee.dev/js/docs/\*\*). When set, only matching URLs are converted.

## `excludeUrlPatterns` (type: `array`):

Glob patterns of URLs to skip (e.g. **/tags/**, \*\*/\*.pdf).

## `maxCrawlDepth` (type: `integer`):

Link depth from start URLs when following links (not used for sitemap URLs).

## `changedSince` (type: `string`):

ISO date (e.g. 2026-09-01). Only sitemap URLs with lastmod on/after this date are crawled; URLs without lastmod are kept.

## `skipUnchanged` (type: `boolean`):

Store a content hash per URL and skip (not output, not billed) pages unchanged since the previous run with the same state key.

## `stateKey` (type: `string`):

Name of the incremental state. Defaults to the first start URL hostname.

## `contentSelector` (type: `string`):

Optional CSS selector of the main content. Otherwise main/article/\[role=main] heuristics are used.

## `removeSelectors` (type: `array`):

Extra CSS selectors removed before conversion (e.g. .cookie-banner).

## `includeHtml` (type: `boolean`):

Also output the cleaned HTML of the main content.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by robots.txt.

## `maxConcurrency` (type: `integer`):

Parallel requests.

## `proxyConfiguration` (type: `object`):

Only needed if the target blocks direct requests (datacenter proxy is usually enough).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxResults": 50,
  "discoveryMode": "sitemap_then_links",
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "maxCrawlDepth": 3,
  "skipUnchanged": false,
  "removeSelectors": [],
  "includeHtml": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "maxResults": 50,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("leadsbrary/sitemap-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "maxResults": 50,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("leadsbrary/sitemap-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxResults": 50,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call leadsbrary/sitemap-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,leadsbrary/sitemap-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xFCP5A5lzCzNONe7h/builds/55FjjhyrOb7EAlujT/openapi.json