# Website to Markdown Crawler for LLM & RAG (`ryumoon-data/website-to-markdown-rag`) Actor

Convert web pages or a whole docs site into clean Markdown for LLMs, RAG and AI agents. Plain HTTP first; headless Chrome only when a page needs JavaScript. Title, metadata and links included. Crawl depth and URL filters. $1 per 1,000 pages; browser pages +$0.80 per 1,000.

- **URL**: https://apify.com/ryumoon-data/website-to-markdown-rag.md
- **Developed by:** [Ryu Moon](https://apify.com/ryumoon-data) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 web pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to Markdown Crawler for LLM & RAG

**Feed any website or docs site to ChatGPT, Claude or your RAG pipeline as clean Markdown, for $1 per 1,000 pages (plus a $0.005 start fee).**

#### Who it's for

AI engineers building RAG and chatbots, developers giving agents web access, and teams turning docs, help centres or pricing pages into a knowledge base.

#### What people use it for

- 📚 **Docs to Markdown for ChatGPT or Claude.** Crawl a documentation section (e.g. `/tutorial/`) and upload it as one knowledge file.
- 🧠 **RAG ingestion.** One item per page with title, canonical URL, language and word count, ready to chunk and embed.
- 💬 **Help-centre chatbots.** Turn support articles into a clean corpus for a customer-support bot.
- 💷 **Competitor pricing pages.** Snapshot pricing pages as Markdown that an LLM can compare.

#### Real sample output

From a real run on `https://docs.python.org/3/tutorial/index.html` (shortened):

```json
{
  "url": "https://docs.python.org/3/tutorial/index.html",
  "title": "The Python Tutorial",
  "siteName": "Python documentation",
  "language": "en",
  "renderedWith": "http",
  "wordCount": 890,
  "markdown": "# The Python Tutorial\n\nTip\n\nThis tutorial is designed for *programmers* that are new to the Python language, **not** *beginners* who are new to programming.\n\nPython is an easy to learn, powerful programming language. It has efficient high-level data structures…"
}
```

#### What it costs (real prices)

Prices are pay per event: **$0.001 per page**, **+$0.0008** only if a page needs headless Chrome, and a **$0.005** start fee per run (1 GB default memory). Failed pages are free.

| Job | Calculation | USD | GBP\* |
|---|---|---|---|
| 100 plain HTML pages | 100 × $0.001 + $0.005 | $0.105 | £0.08 |
| 1,000 plain HTML pages | 1,000 × $0.001 + $0.005 | $1.005 | £0.75 |
| 500-page docs site, 100 need Chrome | 500 × $0.001 + 100 × $0.0008 + $0.005 | $0.585 | £0.44 |

\*GBP at 1 USD = £0.74711 (ECB rate, 6 Nov 2026). Apify bills in USD.

### Integrations

All recipes below use Apify's own official integrations. You need a free Apify account.

- **Zapier.** Add Apify as the action in a Zap, pick **Run Actor** (or **Run Task** for a saved example), choose `ryumoon-data/website-to-markdown-rag` and paste the input JSON. Then add **Fetch dataset items** to pass the results to Google Sheets, Slack, Notion and so on. Synchronous runs in Zapier time out after 30 seconds, so for big batches use the **Finished Actor run** trigger in a second Zap. [Apify docs: Zapier](https://docs.apify.com/platform/integrations/zapier)
- **Make.** Add the Apify module **Run an Actor** with "Run synchronously" set to Yes, then **Get Dataset Items** using the run's default dataset ID, then e.g. Google Sheets "Bulk Add Rows". For long runs, use the **Watch Actor Runs** trigger instead. [Apify docs: Make](https://docs.apify.com/platform/integrations/make)
- **n8n.** Install the Apify node (`@apify/n8n-nodes-apify` from Community Nodes on self-hosted, or search "Apify" in the nodes panel on n8n Cloud). Add an Apify API key or OAuth2 (Cloud) credential, then use it as an action to run this Actor, or as the **On new Apify Event** trigger to process results when a run finishes. [Apify docs: n8n](https://docs.apify.com/platform/integrations/n8n)
- **MCP (Claude, ChatGPT and other AI agents).** Add Apify's hosted MCP server to your client. To load only this Actor, use `https://mcp.apify.com?tools=ryumoon-data/website-to-markdown-rag`. The first time you connect, your browser opens so you can sign in to Apify. Alternatively, send an `Authorization: Bearer <APIFY_TOKEN>` header:
  ```json
  { "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=ryumoon-data/website-to-markdown-rag" } } }
  ```
  Your assistant can then run it from a prompt like *"Convert https://fastapi.tiangolo.com/tutorial/ and its subpages to Markdown"*. [Apify docs: MCP server](https://docs.apify.com/platform/integrations/mcp)

***

**Turn any web page, or a whole docs site, into clean Markdown that LLMs understand.** Give it URLs and get back one item per page: tidy Markdown (headings, lists, tables, code blocks, links), the title, description, author, publish date, language and canonical URL. Menus, cookie banners, footers and ads are stripped out, so you don't waste tokens on boilerplate.

Every page is fetched with **plain HTTP first**, and headless Chrome starts only when a page really needs JavaScript or shows a bot check. You pay **$1 per 1,000 pages**, plus a small extra only for the pages that needed a browser.

### Use with AI agents (MCP, Claude, ChatGPT, Cursor)

**Purpose:** Turn any web page, or a whole website, into clean Markdown text for LLMs, RAG pipelines and AI agents, with title, metadata and links. Renders JavaScript only when needed.

**When to use it**

- The agent needs the readable content of one or more URLs (docs, articles, product pages) as Markdown.
- Building a knowledge base or RAG index from a whole site or a docs section (set `maxCrawlDepth` and `includeUrlPatterns`).

**When not to use it**

- For structured fields such as prices or contact details, use a dedicated scraper (e.g. [Company Contact Finder](https://apify.com/ryumoon-data/company-contact-details-scraper)).
- Pages behind a login are not supported.

**Example input**

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxCrawlDepth": 1,
  "maxPages": 20,
  "mainContentOnly": true
}
```

**Output fields (one dataset item per row):** `url`, `title`, `markdown`, `wordCount`, `description`, `language`, `publishedAt`, `statusCode`, `renderedWith` (http/browser), `error`.

**Cost:** $0.005 per run + $0.001 per page ($1 per 1,000 pages). Cap spend with `maxPages`.

**Connect it to your agent:** add this remote MCP server (Apify's official MCP server, loaded with just this tool):

```json
{"mcpServers": {"website-to-markdown-rag": {"url": "https://mcp.apify.com?tools=ryumoon-data/website-to-markdown-rag"}}}
```

Or call it from any agent with the Apify API: `POST https://api.apify.com/v2/acts/ryumoon-data~website-to-markdown-rag/run-sync-get-dataset-items` with the input above as the JSON body. Agents without an Apify account can pay per run with [agentic payments](https://docs.apify.com/agents/payments) (x402, MPP or Skyfire).

### What can it do?

- 🧹 **Clean Markdown**: main content only (Mozilla Readability), with GitHub-style tables and fenced code blocks.
- ⚡ **HTTP first, browser only when needed**: in `auto` mode Chrome is used only for JavaScript apps (React/Vue/Next shells) and pages with bot checks.
- 🕸️ **Crawl a site**: follow links on the same website to a depth and page limit you choose, with include/exclude regex filters (great for `/docs/` or `/blog/`).
- 🧾 **Metadata for citations**: title, description, author, published/modified date, language, canonical URL, site name and preview image.
- ✂️ **LLM-friendly options**: drop links or images, cap Markdown length per page, remove your own CSS selectors, or get plain text or cleaned HTML instead.
- 🤖 **Works with AI agents**: like every Apify Actor, call it from the Apify API or Apify's official integrations: MCP server, LangChain, LlamaIndex, n8n, Make and Zapier.

### Use cases

- **RAG pipelines**: load documentation, help centres or blogs into a vector database (Pinecone, Qdrant, Weaviate and others).
- **AI agents and chatbots**: give your agent a "read this URL" tool.
- **Fine-tuning and evals**: collect clean text datasets from public pages.
- **Content monitoring and summaries**: fetch articles daily and summarise them with an LLM.
- **SEO and content audits**: word counts, titles, descriptions and canonical URLs in one table.

### How much does it cost?

Pay-per-event pricing. Failed pages are free.

| Event | Price | When it is charged |
|---|---|---|
| Actor start | **$0.005** | Once per run (at the default 1 GB memory) |
| Web page | **$0.001** | Each page converted to Markdown ($1 per 1,000) |
| Browser render | **$0.0008** | Extra, only for pages that needed headless Chrome |

Examples:

- 1,000 normal HTML pages (docs, blogs, news): **about $1.01**
- 100 pages from a JavaScript-heavy app: **about $0.19**
- One quick page for an agent: **$0.006**

Set `renderingMode` to `http` to never pay the browser extra, and use *Max pages* or Apify's *maximum cost per run* to cap spend.

### Input

| Field | What it does | Default |
|---|---|---|
| `startUrls` (required) | Pages to convert | two example pages |
| `renderingMode` | `auto`, `http` or `browser` | `auto` |
| `mainContentOnly` | Strip navigation, footers, sidebars | `true` |
| `outputFormats` | Any of `markdown`, `text`, `html` | `["markdown"]` |
| `includeLinks` / `includeImages` | Keep links / images in Markdown | `true` / `false` |
| `maxMarkdownChars` | Truncate long pages (0 = no limit) | `0` |
| `removeSelectors` | Extra CSS selectors to delete | none |
| `maxCrawlDepth` | 0 = only your URLs; 1+ = follow same-site links | `0` |
| `maxPagesPerStartUrl` / `maxPages` | Crawl limits | `10` / `100` |
| `includeUrlPatterns` / `excludeUrlPatterns` | Regex filters for followed links | none |
| `includePageLinks` | Add a `links` array to each item | `false` |
| `requestTimeoutSecs`, `maxConcurrency`, `proxyConfiguration` | Advanced | `30`, `8`, none |

Crawl the Apify docs' academy section, 50 pages, no links in the Markdown:

```json
{
  "startUrls": [{ "url": "https://docs.apify.com/academy" }],
  "maxCrawlDepth": 2,
  "maxPagesPerStartUrl": 50,
  "includeUrlPatterns": ["/academy/"],
  "includeLinks": false
}
```

### Output

One dataset item per page (Markdown shortened):

```json
{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "finalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "statusCode": 200,
  "title": "Retrieval-augmented generation - Wikipedia",
  "description": null,
  "language": "en",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "renderedWith": "http",
  "depth": 0,
  "markdown": "# Retrieval-augmented generation - Wikipedia\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information...",
  "wordCount": 2218,
  "markdownTruncated": false,
  "crawledAt": "2026-11-02T17:40:00.000Z",
  "error": null
}
```

Pages that fail (404, timeouts, blocked) are still listed with `error` set, and you are not charged for them. A `SUMMARY` record in the key-value store has the run's counts.

Export the dataset as JSON, JSONL, CSV or Excel, or read it straight from the API: `GET https://api.apify.com/v2/datasets/{datasetId}/items?format=json`.

### Using it from an AI agent

- **Apify MCP server**: add this Actor to your MCP client (Claude, Cursor, VS Code and others) and the agent can call it with `{"startUrls":[{"url":"..."}]}`.
- **API, synchronous**: `POST https://api.apify.com/v2/acts/ryumoon-data~website-to-markdown-rag/run-sync-get-dataset-items?token=YOUR_TOKEN` with the input JSON as the body returns the items directly.
- **LangChain / LlamaIndex**: use the `ApifyWrapper` / Apify dataset loader, mapping `markdown` to the document text and `url` to the source.

### Run it from code (Python) and load into LangChain

`pip install apify-client`:

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("ryumoon-data/website-to-markdown-rag").call(run_input={
    "startUrls": [{"url": "https://docs.python.org/3/tutorial/index.html"}],
    "maxCrawlDepth": 1,
    "maxPages": 30,
    "includeUrlPatterns": ["/3/tutorial/"],
    "includeLinks": False,
})
pages = [p for p in client.dataset(run.default_dataset_id).iterate_items() if not p.get("error")]
print(len(pages), "pages")
```

LangChain (`pip install langchain-apify`): load a finished run's dataset with `ApifyDatasetLoader(dataset_id=..., dataset_mapping_function=lambda p: Document(page_content=p["markdown"], metadata={"source": p["url"]}))`. See [Apify docs: LangChain](https://docs.apify.com/platform/integrations/langchain).

### Chunking tips for RAG

- Split on `##` headings first, then by size; keep code fences whole.
- Store `url`, `title` and the heading path as metadata so answers can cite the page.
- Set `includeLinks: false` for shorter embeddings; set `maxMarkdownChars` to cap huge pages.
- Re-run on a schedule (Apify Schedules) to keep the index fresh.

### Tips and limits

- Sites with heavy bot protection (DataDome, PerimeterX, some Cloudflare setups) may still block the request. Those pages are returned with an error and not charged. You can add your own proxy URLs.
- Apify residential proxy is not supported, to keep the per-page price low.
- PDFs, images and other binary files are skipped.
- Please respect each website's terms and robots rules, and only collect content you're allowed to use.

### FAQ

**How is this different from Apify's RAG Web Browser?** RAG Web Browser starts from a Google search and fetches the top results. This Actor converts URLs you already have (or crawls a site from them), doesn't search, and only starts a browser for pages that need one.

**Can I get plain text instead of Markdown?** Yes: set `outputFormats` to `["text"]` (or several formats at once).

**Is it legal?** It reads public web pages like a browser does. You're responsible for how you use the content, including copyright and personal data rules.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to convert. With 'Crawl depth' above 0 the Actor also follows links on the same website.

## `renderingMode` (type: `string`):

auto = fast plain HTTP first, headless Chrome only if the page is a JavaScript app or shows a bot check (recommended). http = never use a browser (cheapest, fastest). browser = always use Chrome.

## `mainContentOnly` (type: `boolean`):

Strip menus, headers, footers, cookie banners and sidebars (Readability). Turn off to keep the whole page body.

## `outputFormats` (type: `array`):

What to return for each page.

## `includeLinks` (type: `boolean`):

Keep [text](url) links. Turn off for shorter LLM context.

## `includeImages` (type: `boolean`):

Keep ![alt](src) image references.

## `maxMarkdownChars` (type: `integer`):

Truncate very long pages to fit your LLM context. 0 = no limit.

## `removeSelectors` (type: `array`):

Extra elements to delete before conversion, e.g. '.related-posts', '#comments'.

## `maxCrawlDepth` (type: `integer`):

0 = only the URLs you give. 1 = also pages linked from them (same website), and so on.

## `maxPagesPerStartUrl` (type: `integer`):

Cap for crawling from each start URL (including itself). 0 = no limit.

## `maxPages` (type: `integer`):

Hard cap for the whole run. 0 = no limit.

## `includeUrlPatterns` (type: `array`):

e.g. '/docs/' or '/blog/'. Applies to crawled links, not your start URLs.

## `excludeUrlPatterns` (type: `array`):

e.g. '/tag/', '?page='.

## `includePageLinks` (type: `boolean`):

Adds a 'links' array (url + anchor text) to each item.

## `requestTimeoutSecs` (type: `integer`):

Give up on a page after this many seconds.

## `maxConcurrency` (type: `integer`):

HTTP pages fetched at once (browser pages are limited to 2 at once).

## `proxyConfiguration` (type: `object`):

Usually not needed. Apify datacenter proxy or your own proxy URLs are supported. Apify residential proxy is not supported.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ],
  "renderingMode": "auto",
  "mainContentOnly": true,
  "outputFormats": [
    "markdown"
  ],
  "includeLinks": true,
  "includeImages": false,
  "maxMarkdownChars": 0,
  "maxCrawlDepth": 0,
  "maxPagesPerStartUrl": 10,
  "maxPages": 5,
  "includePageLinks": false,
  "requestTimeoutSecs": 30,
  "maxConcurrency": 8,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All converted pages in the run's default dataset.

## `summary` (type: `string`):

Counts of converted, browser-rendered and failed pages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
        },
        {
            "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
        }
    ],
    "maxPages": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("ryumoon-data/website-to-markdown-rag").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://docs.apify.com/academy/web-scraping-for-beginners" },
        { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" },
    ],
    "maxPages": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("ryumoon-data/website-to-markdown-rag").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    }
  ],
  "maxPages": 5
}' |
apify call ryumoon-data/website-to-markdown-rag --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ryumoon-data/website-to-markdown-rag"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fnvcZfdvhJbl8kE12/builds/VD9QjuOaxXRUbyYwG/openapi.json