# URL to Markdown for LLMs & RAG: Pages, PDFs & Sitemaps (`cleandata-labs/url-to-markdown`) Actor

Convert web pages and PDFs into clean, LLM-ready Markdown, text or HTML with metadata. Small crawls via links or sitemap, CSS selectors, JS fallback, RAG chunks, respects robots.txt. Pay only per converted page.

- **URL**: https://apify.com/cleandata-labs/url-to-markdown.md
- **Developed by:** [CleanData Labs](https://apify.com/cleandata-labs) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 converted pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown for LLMs & RAG – Pages, PDFs, Sitemaps & Small Crawls

### What does URL to Markdown do?

URL to Markdown is a **web page to Markdown converter and HTML-to-Markdown API** for LLMs, RAG pipelines and AI agents. Give it URLs – web pages, PDFs, a sitemap or a small website section – and it returns the main text as clean Markdown, plain text or HTML, with metadata and optional token-sized chunks.

Turn public web pages **and PDFs** into **clean, LLM-ready Markdown** (or plain text / cleaned HTML) – main content only (no menus, ads or cookie text), with **title, author, publish date, language, description and canonical URL**. Optionally follow links or a **sitemap** to convert a whole docs section, and split the text into **token-sized chunks with overlap** for RAG / vector databases.

Pages are fetched with fast HTTP requests first; a headless browser is used **only when a page needs JavaScript** – so most pages convert in about a second, and you **only pay for pages that were converted**. Errors, blocked pages, robots.txt-disallowed URLs and duplicate pages are free.

### Why use it

- **Main-content extraction** (article mode) or full-page Markdown – your choice; fine-tune with **CSS selectors to keep or remove** elements
- **Output formats:** `markdown`, `text`, `html` – one or several per page
- **PDF to text/Markdown** (up to N pages) – the same price as a web page
- **Small crawls:** follow links up to a depth, or start from the site's **sitemap.xml**, filter with **include/exclude URL globs**, limit pages per start URL; stays on the same host; duplicate pages (same final URL or same content) are skipped and not charged
- **JavaScript pages:** auto browser fallback, optional *wait for selector* and extra wait time
- Keeps **links (absolute URLs), tables and optionally images**; optional **list of internal/external links** per page
- **Chunking for RAG**: chunk size and overlap in approximate tokens, split on paragraph boundaries
- **Respects robots.txt** by default and identifies itself with an honest User-Agent; custom headers supported
- Parallel fetching (default 8), retries with backoff on 429/5xx, optional Apify Proxy
- Works as a tool for AI agents via the Apify API and MCP

### How to use URL to Markdown

1. Open the Actor in Apify Console (click **Try for free** on this page) and sign in or create a free Apify account.
2. Paste the pages or PDF links into **URLs**. To convert a whole section, set **Crawl depth** above 0 or turn on **Use sitemap**, and limit it with **Max pages per start URL (crawl)** and **Include URL patterns** / **Exclude URL patterns**.
3. Pick **Output formats** (`markdown`, `text`, `html`) and an **Extraction mode** (`auto`, `article` for main content only, or `full`).
4. Optional for RAG: set **Chunk size (approx. tokens)** and **Chunk overlap (approx. tokens)**. For JavaScript-heavy sites, use **JavaScript rendering** and **Wait for selector (browser)**.
5. Click **Start**. When the run finishes, open the **Output** tab and download the results as **JSON, CSV, Excel** or HTML.
6. To repeat it, create a **Schedule**, or call the Actor from your app via the Apify API.

### Input examples

Convert a list of pages:

```json
{
  "urls": ["https://react.dev/learn", "https://arxiv.org/pdf/1706.03762"],
  "outputFormats": ["markdown"],
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 80
}
```

Convert a docs section from its sitemap:

```json
{
  "urls": ["https://docs.apify.com/"],
  "useSitemap": true,
  "includeUrlGlobs": ["https://docs.apify.com/academy/*"],
  "maxPagesPerStartUrl": 50
}
```

Follow links two levels deep and drop page furniture:

```json
{
  "urls": ["https://books.toscrape.com/"],
  "maxCrawlDepth": 2,
  "maxPagesPerStartUrl": 100,
  "excludeUrlGlobs": ["*/category/*"],
  "removeElementsCssSelector": "nav, footer, .sidebar"
}
```

### Output example

```json
{
  "url": "https://react.dev/learn",
  "finalUrl": "https://react.dev/learn",
  "statusCode": 200,
  "title": "Quick Start – React",
  "language": "en",
  "publishedAt": null,
  "markdown": "## Quick Start\n\nWelcome to the React documentation! ...",
  "characters": 18342,
  "words": 2711,
  "approxTokens": 4585,
  "fetchedWith": "http",
  "crawlDepth": null,
  "chunks": [{ "index": 0, "text": "## Quick Start ...", "approxTokens": 790 }]
}
```

`fetchedWith` is `http`, `browser` or `pdf`. With `includeLinkList` you also get `links.internal` / `links.external`. Failed URLs come back as rows with `error` (and `blocked: true` for 401/403/429/bot walls) and are not charged.

### How much does it cost?

URL to Markdown uses **pay-per-event pricing: $0.002 per converted page** ($2 per 1,000 pages). Web pages and PDFs cost the same, and you don't pay separately for compute.

Pay per converted page (web page or PDF – same price). Failed, blocked, robots.txt-disallowed and duplicate URLs are free. Set a *maximum cost per run* and the Actor stops cleanly when it is reached – useful for crawls. The only other charge is Apify's standard **Actor start** event ($0.00005 per run, per GB of memory).

Apify's free plan includes **$5 of platform credit every month**, which is enough to try the Actor on real data before you pay anything.

### Use with AI agents (MCP)

URL to Markdown works as a ready-made tool for Claude, ChatGPT, Cursor, VS Code and any other MCP client through the official [Apify MCP server](https://docs.apify.com/platform/integrations/mcp). The agent gets one tool, `cleandata-labs--url-to-markdown`, with the input fields described on this page, and you pay the normal pay-per-event price from your Apify account (a free account works; it includes $5 of monthly credit).

**MCP server URL with only this Actor:**

```text
https://mcp.apify.com?tools=cleandata-labs/url-to-markdown
```

On first connection your client opens the Apify sign-in (OAuth). To use an API token instead, send the header `Authorization: Bearer <YOUR_APIFY_TOKEN>`.

**Claude (claude.ai and Claude Desktop):** Settings → Connectors → Add custom connector. Name: `URL to Markdown`, Remote MCP server URL: the URL above.

**Claude Desktop with a local config file** (`claude_desktop_config.json`, needs Node.js 18+):

```json
{
  "mcpServers": {
    "url-to-markdown": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server", "--tools", "cleandata-labs/url-to-markdown"],
      "env": { "APIFY_TOKEN": "<YOUR_APIFY_TOKEN>" }
    }
  }
}
```

**Cursor** (`~/.cursor/mcp.json`, or `.cursor/mcp.json` in a project):

```json
{
  "mcpServers": {
    "url-to-markdown": {
      "url": "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"
    }
  }
}
```

**Claude Code:**

```bash
claude mcp add --transport http url-to-markdown "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"
```

**VS Code** (`.vscode/mcp.json`):

```json
{
  "servers": {
    "url-to-markdown": { "type": "http", "url": "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown" }
  }
}
```

Already connected to `https://mcp.apify.com` with the default tools? Just ask your agent to use `cleandata-labs/url-to-markdown`; it can run it through `call-actor`. To load several Actors in one connection, separate them with commas, e.g. `https://mcp.apify.com?tools=cleandata-labs/google-trends-explorer,cleandata-labs/url-to-markdown`.

#### Tips for agents

- Smallest useful input: `{"urls": ["https://docs.apify.com/platform/actors"], "maxCharacters": 20000}` (one page as clean Markdown, cut to ~5k tokens).
- A single page usually takes 2-5 seconds; small crawls (10-20 pages) well under a minute, so results are usually ready within the MCP call's default 30-second wait. The call returns a dataset ID; the agent reads the rows with `get-dataset-items`.
- Cost: $0.002 per converted page or PDF. Blocked, 4xx, robots.txt-disallowed and duplicate URLs are free.
- Each row has `approxTokens`, so the agent can decide how much to read. Use `maxCharacters` to keep long pages small.
- Always pass your own `urls`: the default list is only a demo.
- For a whole docs section use `maxCrawlDepth: 1` or `useSitemap: true` with `maxPagesPerStartUrl` (e.g. 10) and `includeUrlGlobs`.
- For RAG set `chunkSizeTokens` (e.g. 800) to get ready-made chunks.

#### Example prompts

1. "Read https://docs.apify.com/platform/actors as Markdown and summarize it in 5 bullet points."
2. "Convert up to 10 pages under https://crawlee.dev/python/docs/ into Markdown chunks of about 800 tokens for my RAG index, and list the page titles."
3. "Extract the text of this PDF: <PDF link>. Read only the first 5 pages and list the key findings."

### Integrations

URL to Markdown works with everything that connects to Apify:

- **Make, Zapier and n8n** – start a run and process the results in your no-code workflows.
- **Google Sheets** – export the dataset to a spreadsheet, or keep a sheet updated with a scheduled run.
- **Webhooks** – get a notification (Slack, email, your own endpoint) when a run finishes or fails.
- **Apify API and clients** – start runs and download results from your code with the REST API, the [Python client](https://docs.apify.com/api/client/python) (`apify-client`) or the [JavaScript client](https://docs.apify.com/api/client/js) (`apify-client`).
- **AI agents** – use it as a tool through the Apify MCP server or the LangChain / LlamaIndex integrations.

Python example:

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("cleandata-labs/url-to-markdown").call(run_input={"urls": ["https://docs.apify.com/platform/actors"], "chunkSizeTokens": 800})
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)
```

### Notes & limitations

- Pages behind logins, paywalls or bot protection are **not bypassed**; they come back as `error` rows (`blocked: true`).
- Crawling is intentionally small-scale (same host, page limits). For crawling entire large websites use a dedicated crawler.
- PDFs: text is extracted (no OCR for scanned PDFs); tables come out as plain text.
- "Tokens" are approximated as characters ÷ 4; exact counts depend on your model's tokenizer.
- You are responsible for complying with each website's terms and copyright when storing or reusing content.

### FAQ

**Is it legal to convert web pages to Markdown?** The Actor only fetches publicly accessible pages, respects robots.txt by default and does not bypass logins or bot protection. Web content can be protected by copyright and site terms, and pages may contain personal data, so you are responsible for making sure your use (storing, training, republishing) complies with the website's terms and applicable law. Ask a lawyer if you are unsure.

**Can I use it with integrations, the API or LLM tools?** Yes. Call it from Python or JavaScript with the Apify API, connect it to Make, Zapier, n8n, Google Sheets or webhooks, or give it to AI agents through the Apify MCP server or LangChain / LlamaIndex. The Markdown and `chunks` fields are ready to send to a vector database.

**Does it work with JavaScript-rendered pages?** Yes. With **JavaScript rendering** set to `auto`, pages that return too little text over plain HTTP are opened in a headless browser. Use `always` for single-page apps and **Wait for selector (browser)** if content loads late.

**Can it convert PDFs?** Yes – text-based PDFs, up to **Max PDF pages** per file, at the same price as a web page. Scanned PDFs (images only) are not OCR'd.

**Why was a URL not charged?** Failed, blocked (401/403/429 or bot walls), robots.txt-disallowed and duplicate pages are returned as `error` rows or skipped, and are free.

# Actor input Schema

## `urls` (type: `array`):

Public web pages or PDF links to convert, e.g. \["https://example.com/blog/post"]. 'https://' is added if missing. One converted page costs one result. Always pass your own URLs: the default list is only a demo.

## `maxCrawlDepth` (type: `integer`):

How far to follow links on the same host, 0-10. 0 (default) = convert only the given URLs. 1 = also the pages they link to, 2 = one level more. Combine with maxPagesPerStartUrl to cap cost.

## `useSitemap` (type: `boolean`):

true = also find pages from the site's sitemap (robots.txt Sitemap entries or /sitemap.xml). Good for a whole docs site or blog. Cap it with maxPagesPerStartUrl. Default false.

## `maxPagesPerStartUrl` (type: `integer`):

Max pages per start URL when crawling or using the sitemap, 0 = no limit. Default 50. Each page is charged, so keep it low (e.g. 5-20) for quick agent tasks.

## `includeUrlGlobs` (type: `array`):

Only crawl URLs that match one of these glob patterns, e.g. \["https://docs.example.com/guide/\*"]. Used only when maxCrawlDepth > 0 or useSitemap is true.

## `excludeUrlGlobs` (type: `array`):

Skip URLs that match any of these glob patterns, e.g. \["*/tag/*", "*/page/*"].

## `extractionMode` (type: `string`):

How much of each page to keep: "auto" (default) = main content, falling back to the full page if too little text is found; "article" = main content only; "full" = whole page including navigation, header and footer.

## `outputFormats` (type: `array`):

One or more of "markdown" (default), "text" (plain text) and "html" (cleaned HTML of the main content). Ask only for what you need: every extra format makes the output larger.

## `removeElementsCssSelector` (type: `string`):

Optional CSS selector of elements to delete before extraction, e.g. ".newsletter, .related-posts, #comments".

## `keepElementsCssSelector` (type: `string`):

Optional CSS selector: if set, only these elements are converted, e.g. "article" or ".docs-content".

## `includeLinkList` (type: `boolean`):

true = also return lists of internal and external links found on each page. Default false.

## `maxPdfPages` (type: `integer`):

PDF links are converted to text automatically. Max pages read per PDF, 0 = all. Default 50.

## `renderJs` (type: `string`):

When to use a headless browser: "auto" (default) = only when the plain HTTP result is too thin; "never" = fastest; "always" = for JavaScript-heavy single-page apps.

## `waitForSelector` (type: `string`):

Optional CSS selector to wait for before reading the page when the browser is used, e.g. "main article".

## `dynamicContentWaitSecs` (type: `integer`):

Extra seconds to wait after page load when the browser is used, 0-30. Default 0.

## `includeLinks` (type: `boolean`):

true (default) = keep hyperlinks as Markdown links with absolute URLs. false = plain text only, which saves tokens.

## `includeImages` (type: `boolean`):

true = keep images as Markdown image tags. Default false.

## `includeTables` (type: `boolean`):

true (default) = convert HTML tables to Markdown tables.

## `chunkSizeTokens` (type: `integer`):

If > 0, also return 'chunks' split on paragraph boundaries for RAG, about this many tokens each (1 token ~ 4 characters), max 32000. Default 0 = no chunks.

## `chunkOverlapTokens` (type: `integer`):

Overlap between consecutive chunks in tokens, 0-2000. Default 50. Used only when chunkSizeTokens > 0.

## `maxCharacters` (type: `integer`):

Cut each page's Markdown to this many characters, 0 = no limit (default). For AI agents a value such as 20000 (~5k tokens) keeps results small enough to read in one step.

## `respectRobotsTxt` (type: `boolean`):

true (default) = skip URLs that the site's robots.txt disallows. Skipped URLs are reported and not charged.

## `minContentCharsBeforeBrowserFallback` (type: `integer`):

In renderJs=auto, use the browser if plain HTTP extraction returns fewer characters than this. Default 400.

## `requestTimeoutSecs` (type: `integer`):

Timeout per attempt in seconds, 5-120. Default 30.

## `maxRetries` (type: `integer`):

Retries per URL on network errors, HTTP 429 and 5xx, 0-5. Default 2.

## `maxConcurrency` (type: `integer`):

Parallel HTTP requests, 1-20. Default 8.

## `proxyConfiguration` (type: `object`):

Optional proxy for sites that block datacenter IPs, e.g. {"useApifyProxy": true}. Default: no proxy.

## `userAgent` (type: `string`):

Optional custom User-Agent header. By default the Actor uses an identifiable bot User-Agent.

## `customHttpHeaders` (type: `object`):

Optional extra HTTP headers, e.g. {"Accept-Language": "de-DE"}. Do not put credentials for sites you are not authorized to access.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Large_language_model",
    "https://docs.apify.com/platform/actors"
  ],
  "maxCrawlDepth": 0,
  "useSitemap": false,
  "maxPagesPerStartUrl": 50,
  "extractionMode": "auto",
  "outputFormats": [
    "markdown"
  ],
  "includeLinkList": false,
  "maxPdfPages": 50,
  "renderJs": "auto",
  "dynamicContentWaitSecs": 0,
  "includeLinks": true,
  "includeImages": false,
  "includeTables": true,
  "chunkSizeTokens": 0,
  "chunkOverlapTokens": 50,
  "maxCharacters": 0,
  "respectRobotsTxt": true,
  "minContentCharsBeforeBrowserFallback": 400,
  "requestTimeoutSecs": 30,
  "maxRetries": 2,
  "maxConcurrency": 8,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Large_language_model",
        "https://docs.apify.com/platform/actors"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("cleandata-labs/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Large_language_model",
        "https://docs.apify.com/platform/actors",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("cleandata-labs/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Large_language_model",
    "https://docs.apify.com/platform/actors"
  ]
}' |
apify call cleandata-labs/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cleandata-labs/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XGn6hv4geVYrkmN6F/builds/bX8fIKzY5CEMpOggW/openapi.json