# URL to Markdown for LLM & RAG (fast, cheap) (`liora-tools/url-to-markdown`) Actor

Convert any public web page to clean, LLM-ready Markdown or text: main-content extraction, metadata, optional same-site crawl and RAG chunking. Plain HTTP, no browser, so it is fast and cheap. Respects robots.txt.

- **URL**: https://apify.com/liora-tools/url-to-markdown.md
- **Developed by:** [Liora Blyth](https://apify.com/liora-tools) (community)
- **Categories:** AI, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown for LLM & RAG: fast, cheap web page to Markdown converter

Convert any **public web page into clean Markdown or plain text** that LLMs, AI agents and RAG pipelines can use
directly. The main content is extracted (menus, cookie banners, footers and ads removed), metadata is added
(title, description, language, published date), and the text can be **pre-chunked for embeddings**.
Optionally **crawl a whole website or docs section** by following same-site links.

It uses plain HTTP with no headless browser, so it is **very fast (about 0.5 s per page) and costs a fraction of
browser-based tools**.

### Use cases

- Feed web pages to ChatGPT, Claude, Gemini, LangChain, LlamaIndex or your own agent
- Build a RAG knowledge base from documentation sites, blogs and help centers
- Monitor articles and turn them into text for summarization or classification
- Website-to-Markdown export for content migration

### Input

```json
{
  "urls": ["https://docs.python.org/3/tutorial/index.html"],
  "extractMode": "main",
  "outputFormat": "markdown",
  "crawlDepth": 1,
  "maxPages": 50,
  "includeUrlRegex": ["/tutorial/"],
  "chunkSize": 2000,
  "chunkOverlap": 200
}
```

- `extractMode`: `main` (article or main content only, the default) or `full` (whole page as Markdown).
- `crawlDepth`: 0 converts only the listed URLs. 1 or more also follows links **on the same website**.
- `includeUrlRegex`: restrict the crawl to matching URLs (for example `/docs/`).

### New options (v0.2)

| Option | What it does |
|---|---|
| `excludeUrlRegex` | Skip crawled links matching a pattern |
| `useSitemap` | Seed the crawl from sitemap.xml (found via robots.txt) |
| `removeSelectors` / `keepSelectors` | Delete nav/footer/cookie banners, or convert only `article`, `main`, ... |
| `includeImages` | Keep images as Markdown image links |
| `saveHtml` | Raw HTML saved to the key-value store (`htmlKey`) |
| `generateLlmsTxt` | Writes `llms.txt` and `llms-full.txt` for the converted pages |
| `maxResults` | Hard stop after N converted pages (cost cap) |
| `requestHeaders` | Extra headers for your own sites (never logged) |
| `onlyChanged`, `stateKey` | Scheduled monitoring: unchanged pages are skipped for free |

Not included on purpose: proxies, "stealth" browsers, CAPTCHA or login bypass. Pages that need JavaScript return `no_text`.

### Output (one item per page)

```json
{
  "url": "https://docs.python.org/3/tutorial/classes.html",
  "status": "ok", "statusCode": 200, "depth": 1,
  "title": "9. Classes", "description": "…", "language": "en", "publishedDate": null,
  "content": "# 9. Classes\n\nClasses provide a means of bundling data and functionality together…",
  "wordCount": 5420, "charCount": 34120,
  "chunks": [{"index": 0, "charStart": 0, "text": "…"}],
  "durationMs": 772
}
```

`status` is `ok`, `no_text` (the page only renders with JavaScript), `skipped` (robots.txt) or `error`.

### Pricing (pay per event)

- **$0.001 per converted page ($1 per 1,000 pages)**, plus Apify's default $0.00005 start fee per run. That is
  cheaper than Web Fetch ($1.50 per 1,000) and RAG Web Browser ($2.28 per 1,000).
- Example: a 300-page documentation site costs about $0.30.
- Errors, robots.txt-skipped pages and pages without text are **free**.

### Performance (measured 2026-10-21)

| Test | Result |
|---|---|
| Wikipedia "Large language model" | 13,058 words, 52 chunks, 1.4 s |
| BBC News front page | 1,298 words, 0.8 s |
| Python tutorial crawl, depth 1, 6 pages | 6 pages in about 4 s total |

### Limits and good behavior

- No JavaScript rendering. Single-page apps that render everything client-side return `no_text` and are not charged.
- robots.txt is respected by default. No captcha or anti-bot bypass and no login.
- Only public `http(s)` URLs are fetched. Private and local network addresses are refused.
- For PDF, Word, PowerPoint and Excel files use our
  [Document to Markdown](https://apify.com/liora-tools/document-to-markdown) Actor.
- You are responsible for having the right to use the content you convert.

# Actor input Schema

## `urls` (type: `array`):

Public web pages to convert.

## `extractMode` (type: `string`):

Main content is best for LLMs and RAG.

## `outputFormat` (type: `string`):

Markdown keeps headings, lists, tables and links.

## `crawlDepth` (type: `integer`):

0 = only the given URLs. 1+ = also follow links on the same website up to this many clicks away.

## `maxPages` (type: `integer`):

Upper limit on pages visited when crawling.

## `includeUrlRegex` (type: `array`):

Optional. When crawling, follow only links that match one of these regular expressions, e.g. /docs/.

## `excludeUrlRegex` (type: `array`):

Crawled links matching any of these regular expressions are skipped, e.g. /tag/ or \\?page=

## `useSitemap` (type: `boolean`):

Also queue pages listed in the site's sitemap (found via robots.txt or /sitemap.xml), up to Max pages. Good for whole-site conversion without deep crawling.

## `chunkSize` (type: `integer`):

If above 0, also split the content into chunks on paragraph boundaries (for embeddings). 0 = no chunking.

## `chunkOverlap` (type: `integer`):

Characters repeated at the start of each next chunk.

## `includeLinks` (type: `boolean`):

Add all absolute links found on the page.

## `includeImages` (type: `boolean`):

Keep images as Markdown image links.

## `removeSelectors` (type: `array`):

Elements to delete before conversion, e.g. nav, footer, .cookie-banner

## `keepSelectors` (type: `array`):

If set, only these elements are converted, e.g. article, main, #content

## `saveHtml` (type: `boolean`):

Store each page's raw HTML in the run's key-value store; the record key is in htmlKey.

## `generateLlmsTxt` (type: `boolean`):

Writes llms.txt (index of pages) and llms-full.txt (all content) to the key-value store at the end of the run.

## `maxResults` (type: `integer`):

Stop after this many converted pages (0 = no limit). Caps your cost.

## `respectRobotsTxt` (type: `boolean`):

Skip pages the site disallows for crawlers (skipped pages are free).

## `acceptLanguage` (type: `string`):

Preferred page language.

## `requestHeaders` (type: `object`):

Extra request headers, e.g. for your own staging site. Never logged. Not for bypassing logins or bot protection.

## `onlyChanged` (type: `boolean`):

For scheduled monitoring: pages whose HTML did not change since the last run with the same state key are skipped for free.

## `stateKey` (type: `string`):

Name of the memory used by 'Only changed pages'.

## `maxConcurrency` (type: `integer`):

How many pages are fetched at the same time.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Large_language_model",
    "https://docs.python.org/3/tutorial/index.html"
  ],
  "extractMode": "main",
  "outputFormat": "markdown",
  "crawlDepth": 0,
  "maxPages": 100,
  "excludeUrlRegex": [],
  "useSitemap": false,
  "chunkSize": 0,
  "chunkOverlap": 200,
  "includeLinks": false,
  "includeImages": false,
  "removeSelectors": [],
  "keepSelectors": [],
  "saveHtml": false,
  "generateLlmsTxt": false,
  "maxResults": 0,
  "respectRobotsTxt": true,
  "acceptLanguage": "en-US,en;q=0.8",
  "requestHeaders": {},
  "onlyChanged": false,
  "stateKey": "default",
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

Web pages converted to Markdown (one dataset item each).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Large_language_model",
        "https://docs.python.org/3/tutorial/index.html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("liora-tools/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Large_language_model",
        "https://docs.python.org/3/tutorial/index.html",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("liora-tools/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Large_language_model",
    "https://docs.python.org/3/tutorial/index.html"
  ]
}' |
apify call liora-tools/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,liora-tools/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vtTxufVE3Mtspae7e/builds/zpjeycV3t3mWruRcc/openapi.json