# URL to Markdown (`munin/url-to-markdown`) Actor

Turn any web page into clean Markdown for LLMs and RAG. Strips navigation, ads and boilerplate, keeps the article. Plain HTTP first, browser only when a page needs it. Charges per page delivered, never for a page it could not read.

- **URL**: https://apify.com/munin/url-to-markdown.md
- **Developed by:** [munin](https://apify.com/munin) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 page delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## URL to Markdown — website content extractor for LLMs and RAG

Give it URLs. Get back the **main content of each page as clean Markdown**, with the navigation, cookie banners, ads, share buttons and footers removed — plus the title, author, publish date, site name and language as separate fields.

It is built for the thing most people actually want a website content crawler for: **filling a RAG index, a vector database or an LLM prompt with readable text.**

- **$2 per 1,000 pages delivered.** Flat. Not compute units, not a monthly rental.
- **Pages it cannot read are free.** A page that 403s, times out, or turns out to be an empty shell is returned as a row with the reason — and is not charged.
- **92% of pages extracted** on a 50-site benchmark spanning docs, news, blogs, forums, recipes, government and Japanese sites. The 4 misses are listed below by name.
- **Fast, because it does not open a browser unless it has to.** 40 of those 50 pages were plain HTTP. A browser was needed for 5.

### What a row looks like

```json
{
  "url": "https://blog.cloudflare.com/workers-ai/",
  "finalUrl": "https://blog.cloudflare.com/workers-ai/",
  "title": "Workers AI: serverless GPU-powered inference on Cloudflare's global network",
  "siteName": "The Cloudflare Blog",
  "author": "Rita Kozlov",
  "publishedAt": "2023-09-27T14:00:00.000Z",
  "language": "en",
  "wordCount": 1830,
  "markdown": "# Workers AI: serverless GPU-powered inference...\n\nToday...",
  "extractedBy": "http/readability",
  "httpStatus": 200,
  "success": true
}
```

`extractedBy` tells you which route produced the row — `http`, `proxy` or `browser`, and whether the body came from the article extractor, from the page's JSON-LD, or from a content container. You can audit any row without re-running it.

### How it gets the content

Three routes, cheapest first. A page stops at the first route that yields real text.

1. **Plain HTTP** with a normal browser's headers. No proxy, so no proxy cost. Most pages end here.
2. **Retry through a proxy**, with a fresh IP — only for pages that answered 401, 402, 403, 406, 429, 451, 5xx, or dropped the connection. Changing IP cannot fix a 404, so a 404 never retries.
3. **A real browser (Chromium)**, only for pages that returned HTML with no readable body — single-page apps that build their content in JavaScript. Images, fonts and video are blocked during the render, which roughly halves the time.

Content selection is Mozilla's Readability — the same extractor behind Firefox Reader View — not a homegrown heuristic. When Readability finds nothing, the actor falls back to the page's JSON-LD `articleBody`, then to `<main>` / `<article>`, in that order.

**A page whose extracted body is shorter than "Minimum words" (default 25) counts as a failure, not as a row.** This is deliberate: returning a Markdown file containing nothing but a nav menu is the single most common complaint about extractors in this category. You are not charged for those.

### Common uses

- **Build a RAG corpus** — point it at a documentation site with "Follow links, how deep" set to 2 and let it walk the docs.
- **Feed a prompt** — turn "Keep links" off so the text going into embeddings is not full of URLs.
- **Watch a set of pages** — schedule it on a fixed URL list and diff `markdown` between runs.
- **Archive articles** — keep `markdown`, `title`, `author` and `publishedAt` as a permanent record.

### Crawling

Set **Follow links, how deep** above 0 and the actor also converts the pages your start URLs link to. It stays on the start URLs' domains unless you turn that off, and you can narrow it further with glob patterns:

| Setting | Example | Effect |
|---|---|---|
| Only follow URLs matching | `https://docs.example.com/guide/**` | Only the guide section is crawled |
| Never follow URLs matching | `**/tag/**` | Tag archive pages are skipped |
| Max pages | `500` | The run stops after 500 pages, successes and failures together |
| Max pages to deliver | `200` | The run stops after 200 *successful* pages, so your cost is capped |

Glob patterns apply to followed links only. Your start URLs are always fetched.

### What it does not do

Written out so you can decide before you spend anything.

- **It does not break bot protection.** Sites behind an aggressive challenge stay unreadable even with residential proxy and a real browser. On the benchmark these were **Reuters, Investopedia, Serious Eats and IMDb** — 4 of 50. They come back as failed rows with the status code, and cost nothing.
- **It does not read PDFs, Word files, images or video.** A non-HTML URL is returned as a failed row saying so. It does not retry them.
- **It does not log in.** No cookies, credentials or session input. Pages behind a login or a hard paywall are not reachable.
- **It does not execute a site's search or fill forms.** It follows `<a href>` links only.
- **It does not render JavaScript by default.** It renders only when plain HTTP gave nothing readable. Set "JavaScript rendering" to *Always* if you know a whole site needs it — slower and slightly more expensive, but not charged differently.
- **It does not guarantee a publish date.** `publishedAt` comes from the page's own metadata. Sites that do not publish one leave the field null; it is never guessed.
- **Infinite scroll is not scrolled.** Only the content present after load is captured.

### Pricing

**$2.00 per 1,000 pages delivered** (`$0.002` per page), charged per successfully extracted page.

Nothing else is charged: no monthly rental, no compute units, no per-run minimum. A page that fails, a page that is too short, and a page that turns out to be a PDF are all free. Use **Max pages to deliver** to put a hard ceiling on a run.

### Benchmark

50 real URLs, run on the Apify platform, 8 in parallel, residential proxy enabled, JavaScript rendering on *Auto*.

| | |
|---|---|
| Pages extracted | **46 / 50 = 92%** |
| Route used | 40 plain HTTP, 1 proxy retry, 5 browser render |
| Wall clock | 32 seconds for all 50 |
| Median body | 711 words |
| Failures | 4, all bot protection, all uncharged |

The URL list is in `test/urls.json` in the actor's source, so the number is reproducible rather than a claim.

### Input in brief

| Field | Default | What it is for |
|---|---|---|
| Start URLs | — | The pages to convert. One per line. |
| Follow links, how deep | 0 | 0 = only your URLs. 1+ crawls onward. |
| Output format | Markdown | Markdown, plain text, cleaned HTML, or all three. |
| Keep links / Keep images | on | Turn off for cleaner embedding input. |
| Remove these elements | — | CSS selectors to delete before extraction. |
| Minimum words | 25 | Below this, the page counts as failed and is free. |
| JavaScript rendering | Auto | Auto / Never / Always. |
| Parallel pages | 8 | Raise for speed, lower to be gentle on a site. |
| Max pages to deliver | 0 | Caps what a run can cost. |

### Notes on responsible use

Crawl what you are allowed to crawl. This actor fetches pages the way a browser does and does not attempt to defeat access controls, log in, or hide what it is. Respect the terms of the sites you point it at, and keep personal data out of what you collect.

# Actor input Schema

## `startUrls` (type: `array`):

The pages to convert. Paste one URL per line. Each one comes back as a row with the page's main content as Markdown, plus title, author, date and language.

## `maxCrawlDepth` (type: `integer`):

0 (default) converts only the URLs above. 1 also converts the pages they link to, 2 goes one step further, and so on. Use with "Max pages" so a crawl cannot run away from you.

## `maxPages` (type: `integer`):

Hard stop on how many pages the run visits, successes and failures together. 0 means no limit.

## `sameDomainOnly` (type: `boolean`):

On by default. Links pointing off the start URLs' domains are skipped. www. is ignored when comparing.

## `includeUrlPatterns` (type: `array`):

Glob patterns, for example https://example.com/docs/\*\*. Leave empty to follow everything allowed by the settings above. Applies to followed links only, never to the start URLs.

## `excludeUrlPatterns` (type: `array`):

Glob patterns to skip, for example **/tag/** or \*\*/?\*. Checked before the include list.

## `outputFormat` (type: `string`):

Markdown is what language models read best and is the default. Plain text drops all formatting. Cleaned HTML keeps the article markup. "All three" puts markdown, text and html on every row.

## `includeLinks` (type: `boolean`):

On keeps Markdown links with absolute URLs. Off keeps the anchor text and drops the URL, which makes cleaner input for embeddings.

## `includeImages` (type: `boolean`):

On keeps image references with absolute URLs. Off removes them.

## `keepNavigation` (type: `boolean`):

Off by default: menus, headers, footers and sidebars are stripped so only the article body is left. Turn on for pages where the list itself is the content.

## `removeSelectors` (type: `array`):

CSS selectors deleted before extraction, for example .cookie-banner or #newsletter-signup. Use this when a site's boilerplate keeps showing up in the output.

## `minWords` (type: `integer`):

A page whose extracted body is shorter than this counts as failed rather than being returned as a near-empty row, and you are not charged for it. Japanese, Chinese and Korean are counted by characters instead.

## `renderJavaScript` (type: `string`):

Auto (default) fetches over plain HTTP first and only opens a browser for pages that return nothing readable. Never skips the browser entirely, which is fastest. Always renders every page in a browser.

## `waitForSelector` (type: `string`):

Only used when a page is rendered in a browser. A CSS selector to wait for before reading the page, for example article or .post-body.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one page before giving up on it. Browser rendering gets 15 seconds more than this.

## `maxConcurrency` (type: `integer`):

How many pages to fetch at once. Higher is faster, but hits per-site rate limits sooner.

## `maxItems` (type: `integer`):

Stops the run once this many pages have been extracted successfully, so you can cap what you spend. 0 means no limit.

## `proxyConfiguration` (type: `object`):

Only used as a retry, for pages that answer 403, 402 or 429 to a direct request. Most pages never touch it. Residential proxy gets through more paywalled-by-IP sites than datacenter.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    },
    {
      "url": "https://blog.cloudflare.com/workers-ai/"
    }
  ],
  "maxCrawlDepth": 0,
  "maxPages": 0,
  "sameDomainOnly": true,
  "outputFormat": "markdown",
  "includeLinks": true,
  "includeImages": true,
  "keepNavigation": false,
  "minWords": 25,
  "renderJavaScript": "auto",
  "requestTimeoutSecs": 30,
  "maxConcurrency": 8,
  "maxItems": 0,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `pages` (type: `string`):

Every row with every field, including the full Markdown of each page.

## `markdownCsv` (type: `string`):

url, title, author, date, word count and the Markdown body, as a spreadsheet.

## `overview` (type: `string`):

The dataset in the Apify console, with the table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
        },
        {
            "url": "https://blog.cloudflare.com/workers-ai/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("munin/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation" },
        { "url": "https://blog.cloudflare.com/workers-ai/" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("munin/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
    },
    {
      "url": "https://blog.cloudflare.com/workers-ai/"
    }
  ]
}' |
apify call munin/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,munin/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gLXWRNsbA7oBUpdeI/builds/uYv9Zszy4DqWFAcMT/openapi.json