# Website to Markdown Crawler for LLM & RAG (llms.txt) (`malmurk/website-to-markdown-crawler`) Actor

Crawl any website and get every page as clean Markdown for LLMs, RAG and AI agents. Main-content extraction, sitemap discovery, RAG chunks, llms.txt and llms-full.txt. Fast HTTP crawler with a flat price per page: no browser, no compute surprises.

- **URL**: https://apify.com/malmurk/website-to-markdown-crawler.md
- **Developed by:** [Catalin Friptu](https://apify.com/malmurk) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLM & RAG (llms.txt)

Crawl a website, a docs section or a blog and get **every page as clean Markdown**, ready for **LLMs, RAG pipelines, vector databases and AI agents**. Menus, cookie banners, sidebars and footers are removed; headings, lists, tables and code blocks are kept. Optional **RAG chunks**, **llms.txt** and **llms-full.txt** come in the same run.

- **Flat, predictable price: $1 per 1,000 pages.** No browser and no compute bill that depends on how heavy a site is. You know the cost before you start: `maxPages` × $0.001.
- **Main-content extraction** with [trafilatura](https://github.com/adbar/trafilatura), plus a fallback to the page's own `<article>`/`<main>` when a page is a structured reference (API docs, changelogs) that generic extractors flatten.
- **Stays where you point it.** `https://example.com/docs/` crawls the docs and nothing else. Glob patterns widen or narrow the scope.
- **Finds pages menus don't link to** by reading the site's `sitemap.xml` (optional).
- **No duplicates.** Redirects, `rel=canonical`, `#fragments` and `utm_*` tracking parameters are collapsed into one page.
- **Polite.** Respects `robots.txt` by default, retries 429/5xx with back-off, sends an honest User-Agent.
- **llms.txt generator.** Get an [llmstxt.org](https://llmstxt.org) index of the site and an `llms-full.txt` with every page's Markdown, free.
- **Failed and empty pages cost nothing.**

### Who it's for

- **AI engineers** loading documentation, help centres or knowledge bases into a vector database for a chatbot or RAG search.
- **Agent and MCP builders** who need "read this site" as one cheap tool call.
- **SEO and content teams** who want a site's text in one file, or an `llms.txt` for their own site.
- **Researchers** archiving blogs and docs as plain text.

### Quick start

1. Paste one or more start URLs, for example `https://docs.example.com/`.
2. Set **Max pages** (you pay only for pages saved, so this also caps the cost).
3. Run. Each page becomes a dataset row with `markdown`; `llms.txt` and `llms-full.txt` land in the run's key-value store.

### Input

| Field | Default | Description |
|---|---|---|
| `startUrls` | | Where the crawl starts. Links are followed only under each start URL's path. |
| `maxPages` | `100` | Stop after this many saved pages. |
| `maxCrawlDepth` | `20` | Links away from a start URL. `0` = only the start URLs. Sitemap pages count as depth 1. |
| `useSitemaps` | `false` | Also queue every in-scope URL from the site's sitemaps (found via `robots.txt`, else `/sitemap.xml`). |
| `includeUrlGlobs` | | Crawl only URLs matching one of these globs, instead of "under the start URL". `**` matches anything, `*` anything but `/`, `{a,b}` alternatives. |
| `excludeUrlGlobs` | | Never crawl URLs matching these globs, e.g. `**/tag/**`. |
| `respectRobotsTxt` | `true` | Skip pages `robots.txt` disallows. Turn off only for your own site. |
| `maxConcurrency` | `10` | Pages fetched at once. |
| `contentMode` | `main` | `main`: article/docs text only. `full`: everything in `<body>`, navigation and footer included. |
| `includeLinks` | `false` | Keep links as `[text](url)`. |
| `includeImages` | `false` | Keep images as `![alt](src)`. |
| `saveText` | `false` | Add a plain-text `text` field. |
| `saveHtml` | `false` | Add the original `html`. |
| `generateLlmsTxt` | `true` | Write `llms.txt` and `llms-full.txt` to the key-value store. |
| `detectLanguage` | `true` | Detect `language` from the text when the page doesn't declare it. |
| `chunkSize` | `0` (off) | Split each page's Markdown into chunks of about this many characters for embeddings. 1000–2000 suits most models. |
| `chunkOverlap` | `200` | Characters shared by neighbouring chunks. |

```json
{
  "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
  "maxPages": 300,
  "useSitemaps": false,
  "excludeUrlGlobs": [{"glob": "**/genindex*"}],
  "chunkSize": 1500,
  "chunkOverlap": 200
}
```

### Output

One row per saved page. A field without a value is `null`.

| Field | Description |
|---|---|
| `url` | The page's URL after redirects. |
| `requested_url` | The URL that was requested, if it redirected. |
| `title`, `description` | From `<title>` and the meta description (Open Graph as fallback). |
| `language` | From `<html lang>`, else detected from the text (ISO 639-1, e.g. `en`, `de`). |
| `author`, `published_at` | From meta tags or `<time datetime>`, when the page has them. |
| `canonical_url` | The page's `rel=canonical`. |
| `depth`, `referrer_url` | Links away from the start URL, and the page the link was found on (`sitemap` for sitemap pages). |
| `http_status` | Usually `200`. |
| `markdown` | The page content as Markdown. |
| `text`, `html` | Only when `saveText` / `saveHtml` is on. |
| `word_count`, `char_count` | Size of `markdown`. |
| `chunks` | When `chunkSize` > 0: `[{"index", "start", "end", "text"}]`, offsets into `markdown`. |
| `truncated` | `true` if a giant page was cut at 4 million characters. |
| `crawled_at` | UTC timestamp. |
| `error` | Only on rows for **start URLs** that failed (`HTTP 404`, `blocked by the site's robots.txt`, `no text content found (the page may need JavaScript to render)`). Dead links found during the crawl are counted in the log instead of filling your dataset. |

Example row from a real run on docs.python.org (`markdown` and `chunks` shortened):

```json
{
  "url": "https://docs.python.org/3/tutorial/classes.html",
  "requested_url": null,
  "title": "9. Classes — Python 3.15.1 documentation",
  "description": "Classes provide a means of bundling data and functionality together. Creating a new class creates a new type of object, allowing new instances of that type to be made. Each class instance can have ...",
  "language": "en",
  "author": null,
  "published_at": null,
  "canonical_url": "https://docs.python.org/3/tutorial/classes.html",
  "depth": 2,
  "referrer_url": "https://docs.python.org/3/tutorial/index.html",
  "http_status": 200,
  "markdown": "# 9. Classes

Classes provide a means of bundling data and functionality together.  Creating
a new class creates a new *type* of object, ...",
  "text": null,
  "html": null,
  "word_count": 5420,
  "char_count": 34733,
  "chunks": [{"index": 0, "start": 0, "end": 1037, "text": "# 9. Classes

Classes provide a means of bundling data ..."}],
  "truncated": false,
  "crawled_at": "2026-10-24T02:58:55+00:00",
  "error": null
}
```

The dataset has four views: **Pages**, **RAG chunks**, **Metadata** and **All fields**. Export as JSON, JSONL, CSV or Excel, or read rows from the API:

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("malmurk/website-to-markdown-crawler").call(run_input={
    "startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 500, "chunkSize": 1500,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page["chunks"] or []:
        print(page["url"], chunk["text"][:80])  # -> your embedding model + vector DB

llms_full = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("llms-full.txt")["value"]
```

#### llms.txt and llms-full.txt

With `generateLlmsTxt` on, the run's key-value store gets two files (links in the run's **Output** tab):

- `llms.txt`: `# Site title`, `> description`, then `- [Page title](url): description` for every saved page, sorted by URL.
- `llms-full.txt`: every page's Markdown in one file, each under `# Title` and `Source: url`. Paste it into a chat, or drop it into a long-context model.

### Pricing

**Pay per event:**

| Event | Price | Charged for |
|---|---|---|
| `page` | **$1 per 1,000 pages** ($0.001 each) | Every page saved to the dataset with content. |
| Actor start | $0.00005 per run | Once per run (per GB of memory). |

- Pages that fail (404, timeouts, DNS errors), are blocked by `robots.txt`, are files (PDF, images, JSON) or have no text **cost nothing**.
- Duplicates (redirects, canonical copies, tracking-parameter variants) are skipped before they are saved, so you never pay twice for one page.
- `llms.txt`, `llms-full.txt`, chunks, language detection and metadata are included.
- The run stops cleanly at the **maximum charge** you set for it.

A 300-page docs site costs **$0.30**. A 5,000-page site costs **$5**.

### FAQ

**Does it render JavaScript?**
No. It reads the HTML the server sends, which is what makes it fast and cheap. Server-rendered sites, static site generators (Docusaurus, MkDocs, Sphinx, Hugo, Jekyll, Next.js/Nuxt with SSR, WordPress) and most docs, blogs and help centres work. A single-page app that renders everything in the browser returns an empty page; you then get an error row for that start URL and pay nothing.

**Why did it stop before `maxPages`?**
Every in-scope page was crawled. Check the scope: `https://example.com/docs/intro` only crawls pages under `/docs/intro`. Start from `https://example.com/docs/` (or the site root) to crawl more, turn on `useSitemaps`, or add `includeUrlGlobs`.

**Can it crawl sites that block bots?**
It does not use proxies or try to bypass blocking, CAPTCHAs or logins. A site that returns 403 to it is respected. Crawl only content you are allowed to use.

**How is this different from a browser-based crawler?**
Browser crawlers bill compute per second of Chrome time, so the same 1,000 pages can cost very different amounts on different sites. This Actor costs $1 per 1,000 saved pages on every site. Measured on Apify: about 1–1.5 pages per second at the default 1 GB memory; more memory crawls faster at the same price.

**Can I keep links and images?**
Yes: `includeLinks` and `includeImages`. They are off by default because plain text embeds better.

**What does main-content mode remove?**
Navigation, headers, footers, sidebars, cookie notices, share buttons and comments. If you need everything on the page, use `contentMode: full`.

**Is there a size limit per page?**
Pages over 3 MB of HTML (rare: giant changelogs and indexes) are skipped and free. Markdown over 4 million characters is cut and flagged `truncated`.

### Limitations

- No JavaScript rendering (see FAQ). No login, cookies or proxies.
- Only HTML pages are converted. Links to PDFs, images, archives and other files are not followed. For PDFs, use [PDF & Document Text Extractor](https://apify.com/malmurk/pdf-text-extractor).
- Main-content detection is heuristic. On unusual layouts compare with `contentMode: full`.
- Code blocks are kept, but blank lines inside them can be dropped.

### How it works

HTTP with [httpx](https://www.python-httpx.org/), main content with [trafilatura](https://github.com/adbar/trafilatura) (Apache-2.0), page conversion with [markdownify](https://github.com/matthewwithanm/python-markdownify) (MIT), `robots.txt` with [Protego](https://github.com/scrapy/protego) (BSD-3), language detection with py3langid (BSD-3).

### Local development

```bash
python -m venv .venv && .venv/Scripts/pip install -r requirements-dev.txt   # Linux/macOS: .venv/bin/pip
python -m pytest -q
docker build -t website-to-markdown-crawler . && docker run --rm -v "$PWD/INPUT.json:/usr/src/app/INPUT.json" website-to-markdown-crawler
```

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts. Only links under each start URL's path are followed: https://example.com/docs/ crawls the docs section, https://example.com/ crawls the whole site.

## `maxPages` (type: `integer`):

Stop after saving this many pages. You pay only for saved pages, so this also caps the cost of a run.

## `maxCrawlDepth` (type: `integer`):

How many links away from a start URL the crawler may go. 0 = only the start URLs themselves. Pages found in sitemaps count as depth 1.

## `useSitemaps` (type: `boolean`):

Also queue every in-scope URL listed in the site's sitemap.xml (found through robots.txt). Finds pages that no menu links to.

## `includeUrlGlobs` (type: `array`):

Crawl only URLs that match one of these patterns, instead of the default 'under the start URL' rule. Example: https://example.com/{blog,docs}/\*\* (\*\* matches anything, \* matches anything except a slash).

## `excludeUrlGlobs` (type: `array`):

Never crawl URLs that match one of these patterns. Example: **/tag/** or https://example.com/blog/page/\*

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site's robots.txt disallows. Turn off only for your own site.

## `maxConcurrency` (type: `integer`):

Pages fetched at the same time. Lower it for small or slow sites.

## `contentMode` (type: `string`):

Main content: the article or documentation text only, without menus, cookie banners, sidebars and footers (best for RAG). Full page: everything in the page body.

## `includeLinks` (type: `boolean`):

Keep hyperlinks as [text](url). Off gives cleaner text for embeddings.

## `includeImages` (type: `boolean`):

Keep images as ![alt](src).

## `saveText` (type: `boolean`):

Add a 'text' field with the content as plain text (no Markdown syntax).

## `saveHtml` (type: `boolean`):

Add an 'html' field with the page's original HTML. Makes the dataset much bigger.

## `generateLlmsTxt` (type: `boolean`):

Write llms.txt (an index of the crawled pages in the llmstxt.org format) and llms-full.txt (every page's Markdown in one file) to the run's key-value store. Free.

## `detectLanguage` (type: `boolean`):

Fill 'language' from the text when the page doesn't declare one in its html lang attribute.

## `chunkSize` (type: `integer`):

Also split each page's Markdown into chunks of about this many characters, ready for embeddings (a 'chunks' field). 0 turns chunking off. 1000-2000 suits most embedding models.

## `chunkOverlap` (type: `integer`):

How many characters neighbouring chunks share, so a sentence cut at a boundary still appears whole in one chunk.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/scraping-basics-javascript"
    }
  ],
  "maxPages": 20,
  "maxCrawlDepth": 20,
  "useSitemaps": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "contentMode": "main",
  "includeLinks": false,
  "includeImages": false,
  "saveText": false,
  "saveHtml": false,
  "generateLlmsTxt": true,
  "detectLanguage": true,
  "chunkSize": 0,
  "chunkOverlap": 200
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `llmsFullTxt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/scraping-basics-javascript"
        }
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("malmurk/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy/scraping-basics-javascript" }],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("malmurk/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/scraping-basics-javascript"
    }
  ],
  "maxPages": 20
}' |
apify call malmurk/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,malmurk/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bNchzVeXqa13Dhadb/builds/Bh5o0PQ0Q7G4fFYRm/openapi.json