# URL to Markdown — Clean LLM-Ready Markdown from Any Web Page (`reverend_sal/url-to-markdown`) Actor

Convert web pages and PDF links to clean Markdown for ChatGPT, Claude, RAG and AI agents. Main-content extraction, metadata, token counts and optional chunks; Chrome only when a page needs JavaScript. From $0.001 per page.

- **URL**: https://apify.com/reverend_sal/url-to-markdown.md
- **Developed by:** [North Eng](https://apify.com/reverend_sal) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown — clean, LLM-ready Markdown from any web page

**URL to Markdown** turns web pages into clean Markdown for LLMs, RAG pipelines, AI agents and note-taking. It strips navigation, cookie banners, ads and footers, keeps the real content (headings, lists, tables, code blocks, links), and returns page metadata plus a token estimate. Optional chunking gives you RAG-ready pieces with their section headings.

It is **HTTP-first**: most pages are fetched and converted without a browser, which is fast and cheap. Only pages that need JavaScript (single-page apps) are rendered in headless Chrome, and you are only billed the browser price when rendering actually produced more content.

### What it does

- **Main-content extraction** (Mozilla Readability, the engine behind Firefox Reader View) with a smart fallback for pages Readability mangles (e.g. big Wikipedia data tables are kept).
- **Proper GFM tables**: real data tables become Markdown tables; layout tables (old-school HTML, Hacker News) are flattened so you don't get junk pipes.
- **Absolute links and image URLs**, so the Markdown works outside the page. Link titles, tracking attributes and `javascript:` links are removed to save tokens.
- **JavaScript pages**: auto-detects client-rendered apps and renders them in Chrome (or force it with `renderMode: "browser"`).
- **PDF links** are converted too (text-based PDFs, page markers included), as are plain-text and native-Markdown URLs (`.md`, `llms.txt`, sites that serve `text/markdown`).
- **Crawl mode**: follow links up to N levels deep, staying under the start URL's path — convert a whole docs section in one run. Respects `robots.txt`.
- **Chunking**: split each page into chunks of ≤ N characters on heading/paragraph boundaries, each tagged with its nearest heading.
- **Metadata**: title, description, author, published date, site name, language, canonical URL, word count, character count, token estimate.

### Why use it

- **RAG and vector databases** — feed clean chunks to Pinecone, Qdrant, Weaviate, pgvector…
- **AI agents and LLM tools** — give your agent a "read this URL" tool that returns compact Markdown instead of 200 KB of HTML (typically 5–20× fewer tokens).
- **Docs ingestion** — crawl `docs.example.com/guide/` into a folder of Markdown files.
- **Content archiving and research** — save articles to Obsidian, Notion or a Git repo.

### How to use

1. Paste one or more **URLs**.
2. Leave **Rendering** on `auto` (recommended).
3. Optional: set **Crawl depth** to 1–2 to convert linked pages, and **Chunk size** (e.g. 2000) for RAG.
4. Click **Start**, then export as JSON/CSV/Excel or call the Apify API. Each page's Markdown is in the `markdown` field.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` | Wikipedia "Markdown" article | Pages to convert. |
| `renderMode` | `auto` | `auto` = HTTP first, Chrome only for JavaScript apps or blocked pages. `http` = never use a browser (cheapest). `browser` = always render in Chrome. |
| `mainContentOnly` | true | Remove navigation, headers, footers, sidebars and cookie banners. Turn off to convert the whole page. |
| `includeLinks` | true | Keep hyperlinks as `[text](url)`. Off = plain text only. |
| `includeImages` | false | Keep images as `![alt](url)`. |
| `chunkSize` | 0 | If > 0, also return `chunks` of at most this many characters (split on headings and paragraphs). |
| `crawlDepth` | 0 | 0 = only the given URLs. 1 = also pages they link to, etc. (same site, max 10). |
| `maxPages` | 50 | Hard cap on pages converted per run (also caps cost). |
| `sameUrlPrefixOnly` | true | When crawling, stay under the start URL's folder (e.g. `/docs/`). |
| `respectRobotsTxt` | true | Skip crawled (linked) pages disallowed by robots.txt. Start URLs you list yourself are always converted. If a site's robots.txt answers with a server error or cannot be fetched, linked pages on that site are skipped (RFC 9309). |
| `maxPdfPages` | 50 | Pages to extract from each linked PDF. |
| `maxConcurrency` | 5 | Pages processed in parallel. |
| `timeoutSecs` | 30 | Per-page timeout. |

Example — convert a docs section for RAG:

```json
{
  "urls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
  "crawlDepth": 1,
  "maxPages": 25,
  "chunkSize": 2000
}
```

### Output

One dataset item per page. Real example (Oct 22 2026, Markdown shortened):

```json
{
  "url": "https://en.wikipedia.org/wiki/Markdown",
  "finalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "statusCode": 200,
  "contentType": "text/html",
  "title": "Markdown - Wikipedia",
  "description": null,
  "author": "Contributors to Wikimedia projects",
  "publishedAt": "2005-08-09T19:56:00Z",
  "siteName": "Wikimedia Foundation, Inc.",
  "language": "en",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Markdown",
  "markdown": "# Markdown - Wikipedia\n\n…\n\n## History\n\nMarkdown was inspired by pre-existing [conventions](https://en.wikipedia.org/wiki/Convention_\\(norm\\)) for…",
  "wordCount": 5297,
  "charCount": 36635,
  "tokensEstimate": 9159,
  "renderedWithBrowser": false,
  "error": null,
  "crawlDepth": 0,
  "convertedAt": "2026-10-22T23:55:31.859Z"
}
```

With `chunkSize` set, each item also has `chunks: [{ "index": 0, "heading": "History", "text": "…", "charCount": 1987 }, …]`. PDF items add `pdfPages` and `pdfPagesExtracted`.

Pages that fail (404, timeout, DNS error, unsupported file type) are listed with `error` and `errorCode` (`HTTP_404`, `TIMEOUT`, `NETWORK_ERROR`, `UNSUPPORTED_TYPE`, `TOO_LARGE`, `BLOCKED_URL`, …) and are **not charged**. Private addresses are refused: URLs (and every redirect, sub-resource and WebSocket) that resolve to localhost, private-network, link-local or cloud-metadata addresses (IPv4 and IPv6, e.g. 169.254.169.254) are not fetched and fail with `BLOCKED_URL`. A run summary is saved to the key-value store as `OUTPUT`.

### Pricing

Pay per event — failed pages are free:

| Event | Price |
|---|---|
| Page converted over HTTP (most pages) | **$0.001** ($1 per 1,000 pages) |
| Page rendered in headless Chrome (JavaScript apps, `renderMode: browser`) | **$0.003** |
| PDF page extracted (from a linked PDF; blank/scanned pages free) | **$0.0005** |
| Actor start | Apify default ($0.00005) |

Example: converting 1,000 normal articles costs about **$1**. In `auto` mode you are only charged the Chrome price when rendering actually returned more content than plain HTTP. Set **Max pages** or a max cost per run to cap spending — the Actor stops cleanly and keeps everything converted so far.

### FAQ

**How is this different from Apify's free Website Content Crawler?**
Website Content Crawler is a full crawler with many knobs and platform-usage pricing. URL to Markdown is a simple, predictable per-page price, HTTP-first (so most runs finish in seconds on 1 GB of memory), with token estimates and RAG chunks built in.

**Does it bypass logins, paywalls or CAPTCHAs?**
No. It converts publicly accessible pages only. Only convert content you are allowed to use.

**Why is my page empty or short?**
Some sites block automated requests or need interaction (cookie walls, infinite scroll). Try `renderMode: "browser"` and `mainContentOnly: false`.

**Scanned PDFs?**
Text is extracted from text-based PDFs; scanned (image-only) pages are returned empty and not charged. For dedicated PDF jobs see our PDF to Markdown Actor.

### Limitations

- No proxy rotation; heavily bot-protected sites may return errors.
- Images are referenced by URL, not downloaded.
- `tokensEstimate` is characters ÷ 4 (a good approximation for English with GPT/Claude-style tokenizers).

# Actor input Schema

## `urls` (type: `array`):

Web pages (or PDF links) to convert. Add a crawl depth below to also convert the pages they link to.

## `renderMode` (type: `string`):

Auto = fast HTTP fetch, and a real Chrome browser only for JavaScript-rendered pages (React/Vue/Next shells) or pages that block plain HTTP. HTTP only = never use a browser (cheapest). Browser = always render in Chrome.

## `mainContentOnly` (type: `boolean`):

Extract the article/main content (Mozilla Readability) and drop menus, headers, footers, cookie banners and sidebars. Turn off to convert the whole page.

## `includeLinks` (type: `boolean`):

Keep hyperlinks as Markdown links with absolute URLs. Turn off for plain text with fewer tokens.

## `includeImages` (type: `boolean`):

Keep images as ![alt](url) with absolute URLs.

## `chunkSize` (type: `integer`):

Also split the Markdown into chunks of at most this many characters on paragraph boundaries, each with its nearest heading — ready for embeddings/RAG. 0 = no chunks. Chunking is free.

## `crawlDepth` (type: `integer`):

0 = only the URLs above. 1 = also convert pages they link to, 2 = links of those, … Only same-site links are followed.

## `maxPages` (type: `integer`):

Upper limit on pages converted in this run (start URLs + crawled pages). Also caps cost.

## `sameUrlPrefixOnly` (type: `boolean`):

When crawling, only follow links under the start URL's folder (e.g. start at https://docs.site.com/guide/ → only /guide/… pages).

## `respectRobotsTxt` (type: `boolean`):

When crawling, skip pages disallowed by the site's robots.txt. Start URLs are always converted.

## `maxPdfPages` (type: `integer`):

PDF links are converted too (charged per PDF page). Stop after this many pages of each PDF.

## `maxConcurrency` (type: `integer`):

Pages processed at the same time.

## `timeoutSecs` (type: `integer`):

Give up on a page after this long.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://en.wikipedia.org/wiki/Markdown"
    }
  ],
  "renderMode": "auto",
  "mainContentOnly": true,
  "includeLinks": true,
  "includeImages": false,
  "chunkSize": 0,
  "crawlDepth": 0,
  "maxPages": 50,
  "sameUrlPrefixOnly": true,
  "respectRobotsTxt": true,
  "maxPdfPages": 50,
  "maxConcurrency": 5,
  "timeoutSecs": 30
}
```

# Actor output Schema

## `pages` (type: `string`):

One item per URL with Markdown, metadata, word/token counts and optional chunks.

## `summary` (type: `string`):

Pages converted, rendered with a browser, failed and skipped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://en.wikipedia.org/wiki/Markdown"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("reverend_sal/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [{ "url": "https://en.wikipedia.org/wiki/Markdown" }] }

# Run the Actor and wait for it to finish
run = client.actor("reverend_sal/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://en.wikipedia.org/wiki/Markdown"
    }
  ]
}' |
apify call reverend_sal/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,reverend_sal/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6eOudzcIhrIEAbJ4u/builds/ENj1uQoIKbJxQ82Rf/openapi.json