# Website to Markdown Crawler for LLM, AI & RAG (`phaux/website-to-markdown-crawler`) Actor

Crawl a website or URL list and return each page's main content as clean Markdown for LLMs and RAG. Use to build a knowledge base from docs or blogs. Input: startUrls, maxPages, optional followLinks, stayUnderStartPath, renderJs. Output per page: url, title, markdown, text, words. Obeys robots.txt.

- **URL**: https://apify.com/phaux/website-to-markdown-crawler.md
- **Developed by:** [Phaux](https://apify.com/phaux) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 converted pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLM, AI & RAG

Convert a **website to markdown** for **LLM** and **RAG** pipelines: a crawler that returns clean, structured text instead of raw HTML.

Point this Actor at a website or a list of pages. It returns each page's main content as clean markdown and plain text, ready for an LLM or a RAG index. It follows links on the same site up to the page limit you set, respects robots.txt, and never uses proxies to get around blocks. Plain HTML pages are cheap, and you pay the higher JavaScript price only when a page really needs a browser. Pages that fail or are blocked are listed and not charged.

### Quick start for AI agents and API users

- **What it does:** Crawl a website or URL list and return each page's main content as clean Markdown for LLMs and RAG.
- **When to use it:** To build a knowledge base from docs or blogs.
- **Example input** (the form's prefill; it finishes in under a minute):

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/"
    }
  ],
  "maxPages": 10,
  "sameDomainOnly": true,
  "renderJs": "never",
  "outputFormats": [
    "markdown",
    "text"
  ]
}
```

- **Example output:** one real dataset row from run `tL1WyvhgDCRCO1BLh` (2026-10-23), with long fields shortened and empty fields left out. Fields not shown are listed in the output table below.

```json
{
  "url": "https://crawlee.dev/",
  "status": 200,
  "depth": 0,
  "renderedWithJs": false,
  "title": "Crawlee · Build reliable crawlers. Fast.",
  "description": "Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for ...",
  "language": "en",
  "canonicalUrl": "https://crawlee.dev/",
  "wordCount": 225,
  "markdown": "Crawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you.\n\n`npx crawlee ...",
  "text": "Crawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you.\nnpx crawlee cr...",
  "crawledAt": "2026-10-23T13:11:44+00:00"
}
```

- **Cost (pay per event, Free-plan prices):** Converted page $0.003; Converted JS-rendered page $0.006. Failed items are not charged.
- **From an AI agent:** with the Apify MCP server, call `call-actor` with actor `phaux/website-to-markdown-crawler`, or load it as its own tool with `https://mcp.apify.com?tools=phaux/website-to-markdown-crawler`. Read results with `get-dataset-items`.

### What does Website to Markdown Crawler do?

- **Crawls a site** from one or more start URLs, following same-site links breadth-first up to **Max pages** and **Max link depth**, or converts just a **URL list** (`followLinks: false`).
- **Extracts the main content** (no menus, footers, cookie banners or ads) with [trafilatura](https://github.com/adbar/trafilatura) and returns **markdown** (headings, lists, tables, code blocks, links) and **plain text**, plus title, meta description, language, canonical URL and word count.
- **Renders JavaScript only when needed:** in `auto` mode a page is opened in headless Chromium only if its HTML has almost no text and looks like a JavaScript app.
- **Polite by design:** robots.txt (including Crawl-delay) is obeyed, one request at a time per host with a delay you set (default 1 s), honest bot User-Agent, no proxies, no CAPTCHA or anti-bot bypass. An HTTP 429 stops requests to that host for the rest of the run.
- **Skips duplicates** (`/page` vs `/page/`, identical content) and non-HTML files.

### Why use it?

- **Fixed price per page**, not pay-per-compute: you know the cost before you run it.
- **Cheap for normal websites:** most pages are plain HTML and are charged at the HTTP price.
- **Clean input for LLMs:** markdown keeps the structure that chunkers and embedding pipelines need.
- **Clear failure list:** every page that failed, was blocked or was skipped is in `FAILED_URLS` with the reason, and none of them is charged.

Use cases: RAG knowledge bases from documentation sites, chatbot training data from your own help center, content audits and migrations, competitive content research on public pages, and feeding pages into GPT/Claude/Llama prompts.

### How to use it

1. Add one or more **Start URLs** (e.g. `https://docs.example.com/`).
2. Set **Max pages** (this caps the cost) and, if needed, **Stay under the start path** or URL patterns.
3. Choose **JavaScript rendering**: `auto` (default), `never` (cheapest) or `always`.
4. Run, then export the dataset as JSON, JSONL, CSV or Excel, or read it via the API.

### Input example

```json
{
  "startUrls": [{"url": "https://crawlee.dev/"}],
  "maxPages": 10,
  "sameDomainOnly": true,
  "renderJs": "never",
  "outputFormats": ["markdown", "text"]
}
```

| Field | Default | Description |
|---|---|---|
| `startUrls` | — | Start pages or a list of URLs. Empty = small example on crawlee.dev. |
| `maxPages` | `50` | Pages saved (and charged) at most. |
| `followLinks` | `true` | Crawl links; off = only the given URLs. |
| `sameDomainOnly` | `true` | Stay on the start URLs' websites. |
| `stayUnderStartPath` | `false` | Stay under the start URL's folder. |
| `maxDepth` | `5` | Link depth from the start pages. |
| `includeUrlPatterns` / `excludeUrlPatterns` | — | Regular expressions to scope the crawl. |
| `renderJs` | `auto` | `auto`, `never` or `always`. |
| `minWordsBeforeJs` | `50` | Auto mode threshold. |
| `outputFormats` | markdown, text | Any of `markdown`, `text`, `html`. |
| `includeLinksInMarkdown` | `true` | Keep `[text](url)` links. |
| `requestIntervalSecs` | `1` | Delay between requests to one host (0.5–30 s). |
| `maxConcurrency` | `4` | Pages in progress at once (one per host at a time). |
| `timeBudgetSecs` | `0` | Stop after N seconds and keep what is saved. |

### Output example

```json
{
  "url": "https://crawlee.dev/python/docs/quick-start",
  "finalUrl": null,
  "status": 200,
  "depth": 1,
  "renderedWithJs": false,
  "title": "Quick start | Crawlee for Python · Fast, reliable Python web crawlers.",
  "description": "...",
  "language": "en",
  "canonicalUrl": "https://crawlee.dev/python/docs/quick-start",
  "wordCount": 954,
  "markdown": "# Quick start\n\nThis short tutorial will help you start scraping with Crawlee in just a minute or two. ...",
  "text": "Quick start\nThis short tutorial will help you start scraping with Crawlee ...",
  "crawledAt": "2026-10-22T01:20:00+00:00"
}
```

`FAILED_URLS` (key-value store) lists pages not saved, with a `code`: `robots_disallowed`, `blocked` (401/403), `rate_limited` (429), `http_error`, `not_html`, `no_content`, `duplicate`, `out_of_scope` or `max_charge_reached`. `RUN_STATS` has the counts.

### Pricing

Pay per saved page:

| Event | Free | Bronze | Silver | Gold+ |
|---|---|---|---|---|
| Converted page (plain HTML) | $0.003 | $0.002 | $0.0017 | $0.0015 |
| Converted JS-rendered page | $0.006 | $0.005 | $0.004 | $0.003 |
| Actor start | $0.00005 | $0.00005 | $0.00005 | $0.00005 |

Example: 1,000 documentation pages (plain HTML) on the Bronze plan cost about **$2.00**. Set a maximum total charge on the run and the crawl stops before it.

### Tips and limits

- Use `renderJs: never` for static sites and docs; it is the cheapest and fastest.
- Large sites: combine **Max pages** with **Stay under the start path** or include patterns.
- Login-only pages, paywalls and pages behind anti-bot walls are not crawled; they show up as `blocked` (free).
- The Actor reads only public pages. You are responsible for following each website's terms and for any personal data on the pages you collect.

### FAQ

**Can I convert only a list of pages without crawling?**
Yes. Put the pages in *Start URLs* and set `followLinks` to `false`.

**Does it render JavaScript?**
In `auto` mode (the default) a page is opened in headless Chromium only when its HTML has almost no text and looks like a JavaScript app. Use `never` for the cheapest runs on static sites, or `always` to render every page.

**How do I limit the cost?**
Set *Max pages* (pages saved and charged at most) and, if you like, a maximum total charge for the run. The crawl stops before it.

**Are failed or blocked pages charged?**
No. They are listed in `FAILED_URLS` with a reason and are free.

**Does it use proxies or get around anti-bot walls?**
No. It obeys robots.txt, sends an honest User-Agent and does not bypass logins, paywalls or CAPTCHAs.

**Which output formats can I export?**
Each page has `markdown` and `text` (and `html` if you ask for it). Export the dataset as JSON, JSONL, CSV or Excel, or read it through the API.

**Something isn't working?**
Open an issue on the Actor's Issues tab with the run ID.

# Actor input Schema

## `startUrls` (type: `array`):

Website start pages or a list of page URLs. Empty = a small example on crawlee.dev.

## `maxPages` (type: `integer`):

Stop after this many pages are saved (= pages you pay for). Default: 50.

## `followLinks` (type: `boolean`):

Crawl links found on the pages. Off = convert only the start URLs (a URL list). Default: true.

## `sameDomainOnly` (type: `boolean`):

Only follow links on the start URLs' websites (www. ignored). Default: true.

## `stayUnderStartPath` (type: `boolean`):

Only follow links under the start URL's folder, e.g. https://example.com/docs/ stays in /docs/. Default: false.

## `maxDepth` (type: `integer`):

Link depth from the start pages (0 = start pages only). Default: 5.

## `includeUrlPatterns` (type: `array`):

Regular expressions; when set, only matching URLs are crawled. Example: /blog/

## `excludeUrlPatterns` (type: `array`):

Regular expressions; matching URLs are skipped. Example: /tag/|/login

## `renderJs` (type: `string`):

auto = render in a browser only when the HTML has almost no text (charged at the JS price); never = plain HTML only (cheapest); always = render every page. Default: "auto".

## `minWordsBeforeJs` (type: `integer`):

In auto mode, a page with fewer words than this whose HTML looks like a JavaScript app is rendered in a browser. Default: 50.

## `outputFormats` (type: `array`):

markdown = main content as markdown; text = plain text; html = raw HTML (large). Example: \["markdown", "text"]. Default: \["markdown", "text"].

## `includeLinksInMarkdown` (type: `boolean`):

Keep [text](url) links in the markdown output. Default: true.

## `requestIntervalSecs` (type: `number`):

Politeness delay per host (0.5–30 s). A longer robots.txt Crawl-delay always wins. Default: 1.

## `maxConcurrency` (type: `integer`):

Pages in progress at once across hosts (1–20). One host is always fetched one request at a time. Default: 4.

## `timeBudgetSecs` (type: `integer`):

Stop after this many seconds and keep the pages saved (0 = only the run timeout guard). Default: 0.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/"
    }
  ],
  "maxPages": 10,
  "followLinks": true,
  "sameDomainOnly": true,
  "stayUnderStartPath": false,
  "maxDepth": 5,
  "renderJs": "never",
  "minWordsBeforeJs": 50,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinksInMarkdown": true,
  "requestIntervalSecs": 1,
  "maxConcurrency": 4,
  "timeBudgetSecs": 0
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `failed` (type: `string`):

No description

## `runStats` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/"
        }
    ],
    "maxPages": 10,
    "sameDomainOnly": true,
    "renderJs": "never",
    "outputFormats": [
        "markdown",
        "text"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("phaux/website-to-markdown-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/" }],
    "maxPages": 10,
    "sameDomainOnly": True,
    "renderJs": "never",
    "outputFormats": [
        "markdown",
        "text",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("phaux/website-to-markdown-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/"
    }
  ],
  "maxPages": 10,
  "sameDomainOnly": true,
  "renderJs": "never",
  "outputFormats": [
    "markdown",
    "text"
  ]
}' |
apify call phaux/website-to-markdown-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,phaux/website-to-markdown-crawler"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Pkvogyjhdp1cNWi90/builds/s0unx5HKrDeiW7Fa6/openapi.json