# Website Content to Markdown Crawler for LLMs (`maliksaid/website-to-markdown`) Actor

Website content crawler: crawl a site and convert pages to clean Markdown for LLMs, RAG and AI training. HTTP only, no browser. Sitemap-first discovery, main content only, tables and code kept. Adds title, language, word count, token estimate, RAG chunks and llms.txt files.

- **URL**: https://apify.com/maliksaid/website-to-markdown.md
- **Developed by:** [Malik Said](https://apify.com/maliksaid) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.20 / 1,000 page conversions

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLMs and RAG

Turn a documentation site, help center or blog into clean Markdown for your LLM, RAG pipeline or AI agent in one run.

- **Main content only.** Menus, footers and cookie notes are removed. Headings, links, tables and code blocks stay.
- **Ready for RAG.** Every page comes with chunks, a heading path and a token estimate, plus `llms.txt` for the whole site.
- **$1.50 per 1,000 pages** on the free plan, less on paid plans. Plain HTTP with no browser. The default 30-page run took 23 seconds on the platform.

One output row, shortened:

```json
{
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start",
  "language": "en",
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two ...",
  "wordCount": 575,
  "tokenEstimate": 4346,
  "chunkCount": 20,
  "chunks": [{ "index": 0, "headingPath": ["Quick Start"], "text": "# Quick Start\n\nWith this short tutorial ...", "tokenEstimate": 73 }]
}
```

The full row is in [Example output](#example-output).

### How the crawler works

This Actor crawls a website and turns each page into clean Markdown for LLMs, RAG pipelines, AI agents and AI training. It uses plain HTTP requests and no browser, so it is fast and cheap. It reads the site's XML sitemaps first, then follows links inside the scope you set. It keeps only the main content of each page, with headings, links, lists, tables and code blocks.

Each page also gets its title, description, language, word count and a token estimate. Optional RAG chunks come with their heading path and token estimate. One run also saves `llms.txt` and `llms-full.txt` files for the site.

Use it on your own website, your documentation, or sites whose content you may reuse.

### What you get

| Field | What it holds |
|---|---|
| `markdown` | Main content as Markdown: headings, links, lists, tables (GitHub style) and fenced code blocks with their language |
| `title`, `description`, `language` | Page title, meta description and the language from the HTML `lang` attribute |
| `wordCount`, `tokenEstimate` | Words in the page and estimated LLM tokens |
| `chunks`, `chunkCount` | Chunks for a vector database, each with `headingPath`, `text`, `tokenEstimate` and `charCount` |
| `contentHash`, `changeStatus` | A hash of the Markdown, and "new" or "changed" in change-only mode |
| `url`, `loadedUrl`, `httpStatus` | Requested URL, URL after redirects, HTTP status |
| `contentSelectorMatched` | Whether your content selector matched on the page. Null when you set none |
| `sourceType` | `html` for web pages, `pdf` for linked PDF files |
| `contentWarning` | `needs-javascript` or `little-text` when a page gave under 50 words and no code block. Null for normal pages |
| `canonicalUrl` | The page's canonical URL, for removing duplicates. Null when the page sets none |
| `metadata` | Site name, keywords, publish and change dates, Open Graph type and image, and the Last-Modified, ETag and Content-Type headers. No author names |
| `depth`, `discoveredBy`, `sitemapLastmod` | Link depth, how the page was found (start, sitemap, link or llms.txt) and the sitemap's last-modified date |

The run's key-value store holds two more files:

- `llms.txt`: an index of all converted pages in the llmstxt.org format, with title, link and description.
- `llms-full.txt`: the Markdown of all pages in one file, each with its source URL. It stops at 20 MB.

### Why use this crawler

- **Price.** $1.50 per 1,000 pages on the free plan, plus a small start fee per run. You pay per page saved, so the cost is known before you start.
- **Speed.** HTTP only, with no browser to start. A 100-page run took 96 seconds on the platform at the default polite rate.
- **Sitemap first.** The crawler reads the XML sitemaps before it follows links. It finds pages that no link points to, and it finds them sooner.
- **RAG chunks in the same row.** No second tool is needed to split pages. A new chunk starts at each H1 to H3 heading and when the size limit is reached. Code blocks and tables stay whole when they fit.
- **llms.txt included.** The same run builds `llms.txt` and `llms-full.txt` for the site.
- **Change-only mode.** Schedule the Actor to keep a knowledge base fresh. It saves only pages that are new or whose content changed.

This crawler does not run JavaScript. Pages that build their content in the browser may come back empty or short. For those sites, use a browser-based crawler.

### How to use it

1. Add one or more start URLs.
2. Choose the crawl scope. By default the crawl stays inside the folder of each start URL. For example, `https://example.com/docs/intro` crawls pages under `https://example.com/docs/`.
3. Set **Maximum pages**. The default is 30, so a first run stays small. Set 0 for no limit.
4. Run the Actor. Download the pages as JSON, CSV or Excel, or read `llms.txt` and `llms-full.txt` from the key-value store.

### Input

| Field | Default | What it does |
|---|---|---|
| Start URLs | crawlee.dev quick start page | Pages where the crawl starts |
| Maximum pages | 30 | Stop after this many pages. Raise it for a full site. 0 means no limit |
| Crawl scope | Same folder | Same folder, same domain, or domain and subdomains |
| Include and exclude URL patterns | none | Globs. `*` matches any text except `/`, `**` matches any text. Include patterns replace the scope |
| Read sitemaps first | on | Queue in-scope sitemap pages before following links |
| Use the site's llms.txt | off | Also crawl the in-scope pages listed in the site's own `/llms.txt` |
| Follow links | on | Follow in-scope links on each page |
| Maximum link depth | 20 | Start URLs are depth 0, sitemap pages depth 1 |
| Main content only | on | Mozilla Readability keeps the main text. Off converts the whole body without header, footer and navigation |
| Content selector | none | CSS selector for the content to convert, such as `article.post`. Pages where it matches nothing use the normal extraction |
| Remove elements | none | CSS selectors to drop before conversion, such as `.ads, #comments` |
| Keep links | on | Links keep absolute URLs. Off keeps only the link text |
| Keep images | off | Images as Markdown image links |
| Include linked PDF files | off | Also convert in-scope linked PDFs to Markdown (text layer, no OCR). Up to 20 MB and 300 pages per file. Each PDF counts as one page |
| Split into chunks for RAG | on | Adds the `chunks` list |
| Chunk size and overlap | 500 and 0 tokens | Largest chunk and the overlap between chunks, in estimated tokens |
| Save llms.txt and llms-full.txt | on | Saves both files to the key-value store |
| Change-only mode and state key | off, "default" | Save only new or changed pages since the last change-only run with the same key |
| Modified since | none | Load only pages changed on or after a date, such as `2026-10-01` or `30 days`. Uses sitemap dates and the If-Modified-Since header. Skipped pages are free |
| Parallel requests | 3 | Pages loaded at the same time |
| Requests per minute | 120 | Rate cap. Requests to one site are spaced evenly |
| Proxy | none | Optional |

### Example input

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "maxPages": 100,
  "scope": "path",
  "useSitemaps": true,
  "chunking": true,
  "chunkSize": 500,
  "saveLlmsTxt": true
}
```

### Example output

One page, with the Markdown and chunk text shortened:

```json
{
  "id": "https://crawlee.dev/js/docs/quick-start",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "loadedUrl": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
  "language": "en",
  "markdown": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the [Introduction](https://crawlee.dev/js/docs/introduction), which is a comprehensive step-by-step guide for creating your first scraper.\n\n## Choose your crawler\n\nCrawlee ...",
  "markdownTruncated": false,
  "wordCount": 575,
  "tokenEstimate": 4346,
  "contentHash": "b1c42a2d0364964c01af5aebb5bed6c0",
  "changeStatus": null,
  "depth": 0,
  "discoveredBy": "start",
  "sitemapLastmod": null,
  "httpStatus": 200,
  "chunkCount": 20,
  "chunks": [
    {
      "index": 0,
      "headingPath": [
        "Quick Start"
      ],
      "text": "Version: 3.18\n\n# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two. To learn in-depth how Crawlee works, read the ...",
      "tokenEstimate": 73,
      "charCount": 291
    },
    {
      "index": 1,
      "headingPath": [
        "Quick Start",
        "Choose your crawler"
      ],
      "text": "## Choose your crawler\n\nCrawlee comes with three main crawler classes: [`CheerioCrawler`](https://crawlee.dev/js/api/cheerio-crawler/class/CheerioCrawler), ...",
      "tokenEstimate": 109,
      "charCount": 433
    }
  ],
  "scrapedAt": "2026-10-22T08:58:58.488Z"
}
```

### Token estimate

`tokenEstimate` counts about 4 characters per token for English and other Latin text, and 2 characters per token for other scripts such as Arabic or Chinese. It is an estimate. Exact counts depend on your model's tokenizer.

### Polite crawling

- robots.txt is read and followed for every page and every sitemap.
- A `Crawl-delay` in robots.txt sets one request at a time with that delay between requests (up to 30 seconds).
- Requests to one site are spaced evenly. The default is 2 requests per second at most.
- Each request has a 45-second timeout and at most 2 retries.

### Pages that are skipped

This Actor never bypasses bot protection, CAPTCHAs or logins. It skips:

- pages with a bot challenge or CAPTCHA wall (for example Cloudflare, DataDome or PerimeterX),
- pages that answer HTTP 401, 403 or 429,
- login pages (a password form with little other text),
- files that are not HTML, such as images, and PDFs unless **Include linked PDF files** is on,
- pages larger than 5 MB, and redirects that leave the crawl scope.

Skipped pages are logged and not charged.

### Pricing

Pay per event. "Page unchanged" applies only in change-only mode. The default run saves up to 30 pages.

| Event | Free plan | GOLD plan and higher |
|---|---|---|
| Run start | $0.005 | $0.005 |
| Page conversion | $0.0015 ($1.50 per 1,000) | $0.0012 ($1.20 per 1,000) |
| Unchanged page checked | $0.001 ($1.00 per 1,000) | $0.001 ($1.00 per 1,000) |

| Pages in one run | Free plan | GOLD plan and higher |
|---|---|---|
| 100 | $0.155 | $0.125 |
| 1,000 | $1.505 | $1.205 |

Costs include the run start fee.
The SILVER plan pays a price between the two columns.

You can set a maximum cost per run in the run options. The crawl stops when it reaches it.

### Change-only mode

Turn on **Change-only mode** and schedule the Actor. The first run saves every page as "new". Later runs with the same state key compare each page's Markdown hash with the last run. Only new and changed pages are saved, with `changeStatus` set. Unchanged pages cost the lower "Page unchanged" price in the table above. The crawler still loads and converts them. `llms.txt` and `llms-full.txt` always list every page converted in the run. Use a different state key for each site you watch.

### Data use

You are responsible for the sites you crawl. Crawl sites you own, or sites whose terms allow reuse of their content. Web pages can hold personal data, such as names in a team page. Handle it under GDPR and other laws that apply to you.

### Limits

- No JavaScript rendering. Pages built by scripts give little text; such rows get `contentWarning: "needs-javascript"`.
- Markdown is cut at 500,000 characters per page (`markdownTruncated` is true).
- Sitemaps: up to 50 sitemap files and 90 seconds of reading. With a page limit, the crawl reads up to twice that many sitemap URLs.

### FAQ

**Can it crawl a whole site?** Yes. Set the scope to "Same domain" and **Maximum pages** to 0.

**Does it work with LangChain, LlamaIndex or a vector database?** Yes. Each chunk has its text and heading path. Load the dataset with the Apify integration for your tool, or download it as JSON.

**How do I remove ads, cookie notes or comments from the Markdown?** Put their CSS selectors in **Remove elements**, for example `.ads, .cookie-notice, #comments`. They are dropped before conversion.

**How do I convert only one part of each page?** Put a CSS selector in **Content selector**, for example `main .article-body`. Only the matched elements become Markdown. Check `contentSelectorMatched` to see where it matched.

**Why is a page short or empty?** The page probably builds its content with JavaScript. This crawler reads the HTML the server sends.

### Related Actors

- [PDF to Markdown and Text Converter](https://apify.com/maliksaid/pdf-to-text-markdown): turn PDF files into Markdown and text, with OCR for scans.
- [Website Screenshot & PDF Capture](https://apify.com/maliksaid/website-screenshot-pdf): full-page screenshots and PDFs of pages that need a browser.
- [HTTP Request Runner](https://apify.com/maliksaid/http-request-runner): call many URLs or APIs in bulk and keep the raw response.
- [SEO Audit & Broken Link Checker](https://apify.com/maliksaid/technical-seo-audit): check the same site for broken links, redirects and missing tags.

# Actor input Schema

## `startUrls` (type: `array`):

Pages where the crawl starts. By default the crawl stays inside the folder of each start URL. For example, https://example.com/docs/intro crawls pages under https://example.com/docs/.

## `maxPages` (type: `integer`):

Stop after converting this many pages. The default keeps test runs short and cheap. Set 0 for no limit.

## `scope` (type: `string`):

Which links the crawl follows. 'Same folder' stays under the start URL's folder. 'Same domain' covers the whole host (www. is ignored). 'Domain and subdomains' also covers blog.example.com and similar.

## `includeGlobs` (type: `array`):

Optional. Crawl only URLs that match one of these patterns. They replace the crawl scope above. \* matches any text except /, \*\* matches any text. Example: https://example.com/docs/\*\*

## `excludeGlobs` (type: `array`):

Optional. Skip URLs that match one of these patterns. Example: https://example.com/**/tag/**

## `useSitemaps` (type: `boolean`):

Read the site's XML sitemaps (from robots.txt, or /sitemap.xml) and queue their in-scope pages before following links. This finds pages faster and finds pages that no link points to.

## `useLlmsTxt` (type: `boolean`):

Also read the site's own /llms.txt and crawl the in-scope pages it links to. robots.txt still applies.

## `followLinks` (type: `boolean`):

Follow in-scope links found on each page. Turn off to convert only the start URLs and sitemap pages.

## `maxCrawlDepth` (type: `integer`):

How many links away from a start URL the crawl may go. Start URLs are depth 0. Sitemap pages are depth 1.

## `mainContentOnly` (type: `boolean`):

Keep only the main article text (Mozilla Readability). Menus, headers, footers and sidebars are removed. Turn off to convert the whole page body without header, footer and navigation.

## `contentSelector` (type: `string`):

Optional. Convert only the elements that match this CSS selector, for example article.post or #content. Pages where it matches nothing use the normal extraction. Leave empty to let the crawler find the main content.

## `removeSelectors` (type: `string`):

Optional. Drop elements that match these CSS selectors before conversion, for example .ads, #comments, .related-posts. Separate selectors with commas.

## `includeLinks` (type: `boolean`):

Keep links as Markdown links with absolute URLs. Turn off to keep only the link text.

## `includeImages` (type: `boolean`):

Keep images as Markdown image links. Off by default, because image links rarely help an LLM.

## `includePdfs` (type: `boolean`):

Also convert in-scope PDF files linked from crawled pages to Markdown (text layer only, no OCR). Each PDF counts as one page. Files over 20 MB, encrypted files and files robots.txt disallows are skipped free. Up to 300 pages of each PDF are read.

## `chunking` (type: `boolean`):

Add a list of chunks to each page, ready for a vector database. Each chunk has its heading path and a token estimate. A new chunk starts at each H1 to H3 heading and when the size limit is reached.

## `chunkSize` (type: `integer`):

Largest chunk, in estimated tokens. The estimate is about 4 characters per token for English.

## `chunkOverlap` (type: `integer`):

Estimated tokens repeated from the end of the previous chunk. At most half the chunk size.

## `saveLlmsTxt` (type: `boolean`):

Save an llms.txt index of all converted pages and an llms-full.txt file with all their Markdown to the run's key-value store. llms-full.txt stops at 20 MB.

## `changeOnly` (type: `boolean`):

Save only pages that are new or whose Markdown changed since the last change-only run with the same state key. Unchanged pages are charged at a lower price. Schedule the Actor with this on to keep a knowledge base fresh.

## `modifiedSince` (type: `string`):

Only load pages changed on or after this date, for example 2026-10-01 or 30 days. Uses the sitemap's last-modified date and the If-Modified-Since header. Start URLs and pages with no date are always loaded. Skipped pages cost nothing.

## `stateKey` (type: `string`):

Name of the memory that change-only mode uses. Use a different key for each site you watch.

## `maxConcurrency` (type: `integer`):

Most pages loaded at the same time. A robots.txt Crawl-delay lowers this to 1.

## `maxRequestsPerMinute` (type: `integer`):

Most page requests per minute across the run. A robots.txt Crawl-delay lowers this. Keep it polite on sites you do not own.

## `proxyConfiguration` (type: `object`):

Optional. Most sites need no proxy. A proxy does not get past bot checks: pages behind a challenge are skipped.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "maxPages": 30,
  "scope": "path",
  "includeGlobs": [],
  "excludeGlobs": [],
  "useSitemaps": true,
  "useLlmsTxt": false,
  "followLinks": true,
  "maxCrawlDepth": 20,
  "mainContentOnly": true,
  "contentSelector": "",
  "removeSelectors": "",
  "includeLinks": true,
  "includeImages": false,
  "includePdfs": false,
  "chunking": true,
  "chunkSize": 500,
  "chunkOverlap": 0,
  "saveLlmsTxt": true,
  "changeOnly": false,
  "stateKey": "default",
  "maxConcurrency": 3,
  "maxRequestsPerMinute": 120,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `llmsFullTxt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/js/docs/quick-start"
        }
    ],
    "maxPages": 30
};

// Run the Actor and wait for it to finish
const run = await client.actor("maliksaid/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "maxPages": 30,
}

# Run the Actor and wait for it to finish
run = client.actor("maliksaid/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/js/docs/quick-start"
    }
  ],
  "maxPages": 30
}' |
apify call maliksaid/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,maliksaid/website-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/p69ZtwGBHcUhZajS9/builds/0eCdhMKnJhyYMqIYM/openapi.json