# URL to Markdown — clean page text for agents, obeys robots.txt (`hermeskd/url-to-markdown`) Actor

Page URLs in, each page's main content out as clean Markdown with title, description, language and word count; a browser renders only when the HTML has too little text. Obeys robots.txt. Pay per page.

- **URL**: https://apify.com/hermeskd/url-to-markdown.md
- **Developed by:** [Logan Parker](https://apify.com/hermeskd) (community)
- **Categories:** Agents, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Give it web page URLs; it returns each page's main content as clean Markdown with title, description, language and word count, rendering the page in a browser only when the plain HTML has too little text. It asks every site for Markdown first, so a page a site publishes as Markdown for agents comes back as the site wrote it — cheaper and truer than extracting it from HTML. It obeys robots.txt, never logs in, never solves CAPTCHAs, and refuses platforms whose terms forbid automated access. PDF and office-file URLs are answered with a pointer to pdf-to-markdown or document-to-markdown. Reading a page is the most common thing an agent does: this Actor is payable per request by an agent, priced per page, and returns only the content — no navigation, footers or cookie banners eating your context.

### Call it from an agent

Through Apify's MCP server (`https://mcp.apify.com`), tool `call-actor`:

```json
{"actor": "hermeskd/url-to-markdown", "input": {"urls": ["https://docs.apify.com/platform/actors/publishing/monetize"]}, "callOptions": {"maxTotalChargeUsd": 0.05, "timeout": 600}, "waitSecs": 30}
```

The call returns the run's status and its `defaultDatasetId`. Read the records with `get-dataset-items`:

```json
{"datasetId": "<defaultDatasetId from the call>", "limit": 50}
```

As a dedicated tool, add `https://mcp.apify.com?tools=hermeskd/url-to-markdown` to your MCP client. A page finishes well inside the 30-second wait, so one `call-actor` plus one `get-dataset-items` is the whole exchange. `maxTotalChargeUsd` caps what the run can charge; when it is reached the remaining URLs come back as `charge_limit_reached` records and are not charged.

### Input

| Field | Type | Required | Default | Meaning |
|---|---|---|---|---|
| `urls` | array of strings | yes | — | Web page URLs, http or https, 1–100 per run. |
| `render` | enum `auto` · `never` · `always` | no | `auto` | `auto` renders a page in a browser only when its HTML yields under 200 words of main content; `never` uses the HTML only; `always` renders every page. |
| `includeLinks` | boolean | no | `false` | Add the page's links as `{text, url}`, absolute and deduplicated, at most 1,000. |
| `maxChars` | integer | no | 200000 | The Markdown is cut beyond this many characters (1,000–1,000,000). |

Every limit above is enforced by the input schema and again in code.

### Output

One record per input URL.

| Field | Meaning |
|---|---|
| `url` | The input URL. |
| `finalUrl` | The page's URL after redirects. |
| `ok` | `true` when the page converted; `false` with `error`. |
| `status` | The final HTTP status; carried on `http_error` records too. |
| `title` | The page's title (its `<title>`, or its `og:title`, or its first heading); for a page served as Markdown, its front matter's title or its first heading. |
| `description` | The page's meta description, when it has one. |
| `lang` | The page's declared language, when it has one. |
| `markdown` | The main content as Markdown — headings, lists, tables and emphasis; navigation, footers and other boilerplate left out; cut at `maxChars`. A page the server sends as Markdown is returned as the site wrote it, its front matter moved into `title`, `description` and `lang`. A plain-text URL is returned as its text, and JSON, XML and CSV as a fenced block. |
| `wordCount` | Words in the Markdown returned. |
| `rendered` | `true` when a browser rendered the page (charged as `page-rendered`), `false` for the HTML alone (`page-converted`). |
| `contentFormat` | What the server sent and the Markdown was made from: `html` (main content extracted), `markdown` (the site's own Markdown), `text`, or `json`, `xml` or `csv`. |
| `links` | With `includeLinks`: the page's links as `{text, url}`. |
| `error` | `{code, message}` when `ok` is `false`. |

Example record for the call above. The docs site answers this Actor with its own Markdown; the record is the one this release returns for the response the site sent on 2026-10-16 (status, final URL and body as served; this Actor's fixture test checks it field by field). The Markdown is shortened here:

```json
{
  "url": "https://docs.apify.com/platform/actors/publishing/monetize",
  "finalUrl": "https://docs.apify.com/actors/publishing/monetize",
  "ok": true,
  "status": 200,
  "title": "Monetize Actors",
  "description": null,
  "lang": null,
  "markdown": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.txt\n> Use this file to discover all available pages before exploring further.\n\n# Monetize Actors\n\nYou can monetize your automation, AI agent, and web scraping projects by publishing them as paid Actors on [Apify Store](https://apify.com/store).\n\n## Pricing models\n\n…",
  "wordCount": 762,
  "rendered": false,
  "contentFormat": "markdown"
}
```

### Events and prices

Pay per event; platform usage is included in the price, never passed through.

| Event | Title | Charged when | FREE / BRONZE | SILVER | GOLD / PLATINUM / DIAMOND |
|---|---|---|---|---|---|
| `page-converted` | Page converted | One page fetched and converted without a browser. | $0.001 | $0.0009 | $0.0008 |
| `page-rendered` | Page rendered and converted | One page rendered in a browser and converted; charged instead of page-converted. | $0.003 | $0.0027 | $0.0024 |

Apify also charges its standard `apify-actor-start` event: $0.00005 per GB of run memory (1 event, $0.00005, at the default 1,024 MB). The minimum maximum-charge a caller can set is $0.01.

Worked example on the FREE tier: 20 documentation pages, 18 read from their HTML and 2 that needed a browser, cost 18 × $0.001 + 2 × $0.003 = $0.024, plus $0.00005 for the start, $0.02405 in all. A page is charged once, as one event or the other; nothing is charged for a failed page. A page that converts but has no text is still charged — the fetch and conversion were done.

### Limits and run time

- 1–100 URLs per run; pages up to 10 MB; the Markdown cut at `maxChars` (200,000 characters by default).
- Each request — its redirects and its whole body included — must finish within 30 seconds (the per-request timeout; a slower page is `download_failed`); at most 2 page loads a second go to one host, whether fetched or rendered. Up to four pages are worked on at once, and the records keep the input order.
- A rendered page is read when its document has loaded and its network has been quiet for half a second, and never more than 5 seconds after the document; one page renders at a time at the default memory (1,024 MB), two at 2,048 MB or more.
- A page the server sends as Markdown (by its content type, or by its body when the type is generic, such as `text/plain`) is never rendered, whatever its length, and is charged as `page-converted`. Every request asks for Markdown first: `Accept: text/markdown, text/html;q=0.9, application/xhtml+xml;q=0.9, text/plain;q=0.8, */*;q=0.5`.
- With `render: always`, a URL that turns out to be a file or plain text, not an HTML page, is answered as a plain fetch answers it and charged as `page-converted` (or not at all, for a file another Actor converts). With `render: auto`, a render that fails for any reason but robots.txt, the deny-list or a sign-in wall returns the page read from its HTML, charged as `page-converted`.
- `noindex` and other indexing directives govern search engines, not fetching: such a page is converted. robots.txt is what decides whether a page may be fetched.
- Typical run time: a static page in about a second, a rendered page in a few seconds, at the default memory (1,024 MB). The default run timeout is 600 seconds.

### Errors

Errors are records with `ok: false` and `error.code`; none is charged.

| `error.code` | Meaning |
|---|---|
| `download_failed` | The page could not be fetched (network error, timeout, or the browser could not load it). |
| `http_error` | The final HTTP status was not 2xx; `status` carries it. |
| `unsupported_content_type` | Not a web page — PDF, office, image, audio and video URLs name the Hermes Actor that converts them, under every `render` mode; legacy .doc, .xls and .ppt files are not read by any Hermes Actor. |
| `too_large` | Larger than 10 MB. |
| `robots_disallowed` | robots.txt forbids fetching the URL, or a page tried to navigate to a URL it forbids. |
| `blocked_host` | The host is on the deny-list, or resolves to a private or local address. |
| `login_required` | The page requires sign-in (HTTP 401, or a sign-in wall); nothing was attempted. |
| `timeout` | The page did not finish within the run's time budget. |
| `internal_error` | Unexpected failure; message carries the class. |
| `charge_limit_reached` | The run reached its maximum total charge before this URL; it was not processed. |
| `invalid_input` | The run input passed the input schema but cannot be used (for example no URL at all); the message names the field. One record per input given, or one with `url: null` when none was given; nothing is processed. |

The run fails only when every input failed with an Actor-side code (`internal_error`, `timeout`), so a broken Actor is visible. Every other run succeeds — including one where nothing converted because of the caller's own inputs (bad URL, robots, deny-list, a sign-in wall, invalid input, the charge limit), or where those and Actor-side failures are mixed.

### Sources and terms

Public pages the caller names. Process only pages you have the right to process. Fetching follows the Hermes fetch policy: robots.txt is obeyed for the user-agent `HermesBot` (and `*`), on every redirect and, in the browser, for every frame's document, and a disallowed URL is refused without a request; platforms whose terms forbid automated access — among them facebook.com, instagram.com, tiktok.com, youtube.com, linkedin.com, x.com, reddit.com, amazon, indeed and Google Search — and people-search sites are on a deny-list and refused without a request, also for anything a rendered page loads; private and local addresses are refused, after DNS and on every redirect; the user-agent is `HermesBot/1.0 (+https://apify.com/hermeskd)` (a rendering browser keeps its own string with `HermesBot` appended); no login, cookies, credentials, CAPTCHA solving, evasion or proxies are used. The page's author fields are never returned.

### Not affiliated

This Actor is not affiliated with Apify's own page fetchers or with any site it reads. trafilatura, Beautiful Soup, lxml, Playwright and Chromium are used under their open-source licences.

### Related Hermes Actors

- `pdf-to-markdown` — PDF documents to Markdown, with OCR for scanned pages.
- `document-to-markdown` — Word, PowerPoint, Excel and other office files to Markdown.
- `image-to-text` — OCR for images.
- `audio-to-transcript` — speech to text.
- `video-frames` — frames and metadata from a video.
- `image-upscale` — image upscaling.
- `audio-video-convert` — audio and video format conversion.

### Changelog

- 1.0 (2026-10-16) — every request asks for Markdown first; a page the server sends as Markdown is returned as the site wrote it, its front matter in `title`, `description` and `lang`, never rendered; new field `contentFormat`.
- 1.0 (2026-10-16) — files under `render: always` are answered like a plain fetch instead of rendered as empty pages; any failed render under `render: auto` keeps the static page; legacy Office files get no pointer.
- 1.0 (2026-10-16) — first release: main content as Markdown, a browser only when needed, links on request, robots.txt for every frame.

# Actor input Schema

## `urls` (type: `array`):

Web page URLs, http or https, 1–100 per run. Each URL becomes one record.

## `render` (type: `string`):

auto renders a page in a browser only when its HTML yields under 200 words of main content; never uses the HTML only; always renders every page (charged as page-rendered).

## `includeLinks` (type: `boolean`):

Add the page's links as {text, url}, absolute and deduplicated, at most 1,000.

## `maxChars` (type: `integer`):

The Markdown is cut beyond this many characters (1,000–1,000,000).

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/platform/actors/publishing/monetize"
  ],
  "render": "auto",
  "includeLinks": false,
  "maxChars": 200000
}
```

# Actor output Schema

## `results` (type: `string`):

One record per input URL with ok, finalUrl, status, title, markdown, wordCount, rendered and error.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/platform/actors/publishing/monetize"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hermeskd/url-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com/platform/actors/publishing/monetize"] }

# Run the Actor and wait for it to finish
run = client.actor("hermeskd/url-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/platform/actors/publishing/monetize"
  ]
}' |
apify call hermeskd/url-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hermeskd/url-to-markdown"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/AUDpfzAWxcL2VD6fP/builds/ZwVYBCCUZRx1pOdV9/openapi.json