# Website to Markdown Crawler for LLM & RAG (Sitemaps, JS) (`datlab/site-to-markdown-crawler`) Actor

Crawl a whole website into clean Markdown and text for LLMs, RAG and vector databases. Fast HTTP crawling with an automatic headless-browser fallback for JavaScript pages. Sitemaps, llms.txt, globs, depth limits. Respects robots.txt.

- **URL**: https://apify.com/datlab/site-to-markdown-crawler.md
- **Developed by:** [DatLab](https://apify.com/datlab) (community)
- **Features:** Limited permissions
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 page crawled (http)s

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLM & RAG (sitemaps, JavaScript)

This Actor is a **website content crawler** that turns a whole site, or just its documentation, blog or help center, into **clean Markdown and text, one item per page**. It is built to crawl documentation for RAG pipelines, LangChain, LlamaIndex, vector databases (Pinecone, Qdrant, Weaviate) and LLM datasets. Unlike pay-per-usage crawlers, you pay a fixed price per page, with platform usage included.

- **Cheap by default.** `auto` mode crawls with fast plain HTTP and renders **only** client-side (JavaScript) pages with headless Chromium. Static pages are billed at the HTTP price.
- **Stays where you point it.** Same host and the start URL's path by default, or your own include/exclude globs, plus max depth, max pages and max results.
- **Finds more pages.** Optional sitemap discovery (robots.txt `Sitemap:` and `/sitemap.xml`, including indexes and .gz), `/llms.txt` and Markdown files.
- **Clean content.** Mozilla Readability, removal of navigation, footers, cookie banners and dialogs, optional cross-page duplicate-line pruning, canonical-URL de-duplication.
- **Extras.** Save cleaned HTML (dataset or file), links, screenshots of browser pages, and same-site documents (PDF, DOCX, XLSX…).
- **Polite.** Always respects robots.txt and Crawl-delay, per-host rate limiting, and treats 401/403/429 as a no rather than something to get around.

### Use cases

- Turn product documentation into a knowledge base for a support chatbot.
- Index a blog or help center into a vector database (Pinecone, Qdrant, Weaviate) or a LangChain / LlamaIndex pipeline.
- Build a Markdown corpus of a site for LLM fine-tuning or evaluation.
- Keep an AI assistant up to date by re-crawling docs on a schedule.

### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | – | Where to start. Scope = same host + URLs whose path starts with the start URL's path. |
| `crawlerType` | `auto` | `auto` (HTTP, browser only when needed), `cheerio` (HTTP only), `playwright:chromium` (browser for every page). |
| `includeUrlGlobs` / `excludeUrlGlobs` | – | Glob patterns, e.g. `https://example.com/{docs,blog}/**`, `**/tag/**`. |
| `maxCrawlDepth`, `maxCrawlPages`, `maxResults` | 20, 1000, unlimited | Limits (also cap your cost). |
| `useSitemaps`, `useLlmsTxt` | `false` | Extra URL discovery. |
| `keepUrlFragments`, `ignoreCanonicalUrl` | `false` | URL identity rules. |
| `htmlTransformer`, `readableTextCharThreshold` | `readableTextIfPossible`, 100 | Main-content extraction. |
| `removeElementsCssSelector`, `keepElementsCssSelector`, `removeCookieWarnings`, `aggressivePrune` | defaults | Content cleanup. |
| `dynamicContentWaitSecs`, `waitForSelector`, `softWaitForSelector`, `maxScrollHeightPixels`, `clickElementsCssSelector`, `blockMedia` | 10 s, –, –, 5000, –, `true` | Browser rendering controls. |
| `saveMarkdown`, `saveText`, `saveHtml`, `saveHtmlAsFile`, `saveLinks`, `saveScreenshots`, `saveFiles` | md + text on | What to store. |
| `maxConcurrency`, `minDelayPerHostMs`, `requestTimeoutSecs`, `maxRequestRetries` | 10, 250, 60, 3 | Speed and politeness. |
| `storeSkippedUrls`, `debugMode`, `proxyConfiguration` | off | Diagnostics and optional datacenter proxy. |

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/academy" }],
  "maxCrawlPages": 200,
  "useSitemaps": true,
  "excludeUrlGlobs": [{ "glob": "**/changelog/**" }]
}
```

### Output

```json
{
  "url": "https://crawlee.dev/js/docs/quick-start",
  "crawl": { "loadedUrl": "https://crawlee.dev/js/docs/quick-start", "loadedTime": "2026-11-11T20:30:21.341Z", "referrerUrl": null, "depth": 0, "httpStatusCode": 200, "renderedWith": "raw-http" },
  "metadata": { "title": "Quick Start | Crawlee for JavaScript", "description": "With this short tutorial…", "languageCode": "en", "canonicalUrl": "https://crawlee.dev/js/docs/quick-start", "author": null, "keywords": null, "publishedAt": null, "modifiedAt": null, "openGraph": { "title": "…" }, "jsonLd": [] },
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping…",
  "text": "Quick Start\n\nWith this short tutorial…",
  "wordCount": 756,
  "tokenEstimate": 2373
}
```

Optional fields: `html`, `htmlUrl`, `links`, `screenshotUrl`. Saved documents appear as `{ "type": "file", "fileUrl": "…", "contentType": "application/pdf", "sizeBytes": … }`. `RUN_SUMMARY` holds the counts and `SKIPPED_URLS` the skipped URLs (if enabled).

### Pricing

Pay per event, with Apify platform usage included, so there are no compute-unit surprises.

| Event | Price | When |
|---|---|---|
| Page (HTTP) | **$0.10 per 1,000** | Each page crawled with plain HTTP and saved |
| Page (browser) | **$0.35 per 1,000** | Each page that needed a headless browser for JavaScript |
| File | **$0.50 per 1,000** | Each linked document saved (only with `saveFiles`) |

Failed pages, duplicates, robots.txt-blocked and skipped URLs are **not charged**. Apify adds a tiny per-run start fee. Use `maxCrawlPages` or **Maximum cost per run** to cap spending.

Example: a 5,000-page documentation site in `auto` mode, mostly static, costs about $0.50.

### FAQ

**How is this different from Website Content Crawler?** Similar inputs and output, but a fixed price per page instead of compute units, and a browser only for pages that need it. Static pages cost $0.10 per 1,000 and JavaScript pages $0.35 per 1,000, all usage included.

**My site is a React/Vue/Angular app.** `auto` detects client-rendered pages and renders them. If a site is fully client-side, choose `playwright:chromium`.

**Why did it stop at N pages?** Check `maxCrawlPages`, `maxCrawlDepth`, the path scope (start URL path), robots.txt, and **Maximum cost per run**.

**Can I crawl sites that need a login?** No. Cookies and auth headers aren't supported, and the crawler never bypasses logins, paywalls or captchas.

**Can I turn off robots.txt?** No, it's always respected.

### Legal & data use

- Crawls only publicly accessible pages, obeys robots.txt and `Crawl-delay`, rate-limits per host and never bypasses logins, captchas, paywalls or other access controls.
- No personal-data extraction features (no email or phone harvesting). Output is page content and metadata.
- You're responsible for crawling only sites you're allowed to, and for using the content (copyright, site terms, applicable law), including for AI training or retrieval.

### Running on Apify

You can run this Actor on the Apify platform from the Console, the API, a schedule, integrations (webhooks, Slack, Google Sheets, Make, Zapier) or MCP.

Want code? There's a walkthrough with JS and Python examples, plus chunking and embedding: [docs site to Markdown for RAG on dev.to](https://dev.to/datlab/turning-a-docs-site-into-clean-markdown-for-rag-for-about-10-cents-per-1000-pages-1nfp).

To use it from Claude, Cursor or another MCP client, add Apify's hosted MCP server and pin this actor: `https://mcp.apify.com/?tools=fetch-actor-details,datlab/site-to-markdown-crawler`

### Other DatLab actors

- [Web Page to Markdown](https://apify.com/datlab/web-page-to-markdown) — turn a list of URLs into Markdown for LLMs and RAG
- [Sitemap Change Monitor](https://apify.com/datlab/sitemap-change-monitor) — extract sitemap URLs and watch new, removed and updated pages
- [ATS Job Listings](https://apify.com/datlab/ats-job-listings) — Personio, Teamtailor, Workable and Recruitee, one schema
- [SAM.gov Contract Opportunities](https://apify.com/datlab/sam-gov-contract-opportunities) — US federal opportunities from the official daily extract, no API key
- [Bulk Website Screenshot](https://apify.com/datlab/bulk-website-screenshot) — full page or mobile screenshots of a URL list, PNG, JPEG or PDF

# Actor input Schema

## `startUrls` (type: `array`):

Where to start. The crawler follows links on the same host whose path starts with a start URL's path (e.g. https://example.com/docs crawls /docs/... only), unless you set include globs.

## `crawlerType` (type: `string`):

Auto crawls with plain HTTP and renders only client-side (JavaScript) pages with Chromium. You pay the browser price only for those pages.

## `includeUrlGlobs` (type: `array`):

Optional. Only follow links matching at least one glob, e.g. https://example.com/{docs,blog}/\*\*. Overrides the default start-URL path scope.

## `excludeUrlGlobs` (type: `array`):

Never follow links matching these globs, e.g. **/tag/** or \*\*/*?page=*.

## `maxCrawlDepth` (type: `integer`):

How many links away from the start URLs to go. 0 = start URLs only.

## `maxCrawlPages` (type: `integer`):

Maximum pages to crawl (including duplicates and failed pages). Caps your cost.

## `maxResults` (type: `integer`):

Stop after saving this many results.

## `useSitemaps` (type: `boolean`):

Also discover pages from the site's sitemaps (robots.txt Sitemap entries and /sitemap.xml). Only in-scope URLs are used.

## `useLlmsTxt` (type: `boolean`):

Fetch /llms.txt if the site has one and follow the Markdown files it links. Markdown is returned as-is.

## `keepUrlFragments` (type: `boolean`):

Treat example.com/#a and example.com/#b as different pages (for hash-routed sites).

## `ignoreCanonicalUrl` (type: `boolean`):

By default, pages with the same rel=canonical URL are saved once. Turn on to save every URL.

## `maxConcurrency` (type: `integer`):

Parallel HTTP requests (browser pages are capped at 3).

## `minDelayPerHostMs` (type: `integer`):

Politeness: minimum gap between requests to the same host. robots.txt Crawl-delay is honored when larger.

## `requestTimeoutSecs` (type: `integer`):

Maximum time to load one page.

## `maxRequestRetries` (type: `integer`):

Retries on network errors and 5xx responses.

## `dynamicContentWaitSecs` (type: `integer`):

For browser-rendered pages: maximum wait for the network to go idle.

## `waitForSelector` (type: `string`):

Optional. The page fails if this element doesn't appear.

## `softWaitForSelector` (type: `string`):

Optional. Wait for this element, but continue if it doesn't appear.

## `maxScrollHeightPixels` (type: `integer`):

Scroll down this far to trigger lazy-loaded content. 0 disables scrolling.

## `clickElementsCssSelector` (type: `string`):

Click matching elements (up to 50 per page) to expand collapsed content, e.g. \[aria-expanded="false"].

## `removeCookieWarnings` (type: `boolean`):

Strip common cookie-consent banners from the output. Nothing is clicked or accepted.

## `blockMedia` (type: `boolean`):

Faster, cheaper rendering.

## `keepElementsCssSelector` (type: `string`):

Optional, e.g. article, main .content.

## `removeElementsCssSelector` (type: `string`):

Removed before conversion. Empty = default (nav, footer, scripts, dialogs, buttons...).

## `htmlTransformer` (type: `string`):

Readable text uses Mozilla Readability. 'If possible' falls back to the whole cleaned page when Readability finds too little.

## `readableTextCharThreshold` (type: `integer`):

Shorter Readability results count as failed.

## `aggressivePrune` (type: `boolean`):

Drop lines already seen on earlier pages (repeated headers, sidebars, banners). Useful for RAG.

## `saveMarkdown` (type: `boolean`):

Include a markdown field.

## `saveText` (type: `boolean`):

Include a text field.

## `saveHtml` (type: `boolean`):

Include an html field (can make the dataset large).

## `saveHtmlAsFile` (type: `boolean`):

Store HTML as a file and add htmlUrl to each item.

## `saveLinks` (type: `boolean`):

Include all links found on each page.

## `saveScreenshots` (type: `boolean`):

Store a viewport JPEG for browser-rendered pages and add screenshotUrl.

## `saveFiles` (type: `boolean`):

Download same-site PDF, DOC(X), XLS(X), PPT(X), CSV, TXT, MD and JSON files linked from crawled pages (max 50 MB each) to the key-value store. Each saved file is charged as a file event.

## `storeSkippedUrls` (type: `boolean`):

Save URLs skipped because of robots.txt or the page limit to the SKIPPED_URLS record.

## `proxyConfiguration` (type: `object`):

Optional. Datacenter proxies only; residential and SERP groups are not supported.

## `debugMode` (type: `boolean`):

Add response headers and page signals to each item.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "crawlerType": "auto",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "maxCrawlDepth": 20,
  "maxCrawlPages": 1000,
  "maxResults": 10000000,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "maxConcurrency": 10,
  "minDelayPerHostMs": 250,
  "requestTimeoutSecs": 60,
  "maxRequestRetries": 3,
  "dynamicContentWaitSecs": 10,
  "maxScrollHeightPixels": 5000,
  "clickElementsCssSelector": "[aria-expanded=\"false\"]",
  "removeCookieWarnings": true,
  "blockMedia": true,
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], form[role=\"search\"], button",
  "htmlTransformer": "readableTextIfPossible",
  "readableTextCharThreshold": 100,
  "aggressivePrune": false,
  "saveMarkdown": true,
  "saveText": true,
  "saveHtml": false,
  "saveHtmlAsFile": false,
  "saveLinks": false,
  "saveScreenshots": false,
  "saveFiles": false,
  "storeSkippedUrls": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "debugMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

One item per crawled page with Markdown, text and metadata.

## `runSummary` (type: `string`):

RUN_SUMMARY record with page, browser, file, duplicate and robots counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        }
    ],
    "clickElementsCssSelector": "[aria-expanded=\"false\"]",
    "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], form[role=\"search\"], button"
};

// Run the Actor and wait for it to finish
const run = await client.actor("datlab/site-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy/web-scraping-for-beginners" }],
    "clickElementsCssSelector": "[aria-expanded=\"false\"]",
    "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [aria-modal=\"true\"], form[role=\"search\"], button",
}

# Run the Actor and wait for it to finish
run = client.actor("datlab/site-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "clickElementsCssSelector": "[aria-expanded=\\"false\\"]",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, iframe, template, canvas, img[src^='\''data:'\''], [role=\\"alert\\"], [role=\\"banner\\"], [role=\\"dialog\\"], [role=\\"alertdialog\\"], [aria-modal=\\"true\\"], form[role=\\"search\\"], button"
}' |
apify call datlab/site-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datlab/site-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0vxDdl65Ec4qxp6DS/builds/z9xouIp3hqKn5g4ax/openapi.json