URL to Markdown — clean page text for agents, obeys robots.txt

Give it web page URLs; it returns each page's main content as clean Markdown with title, description, language and word count, rendering the page in a browser only when the plain HTML has too little text. It asks every site for Markdown first, so a page a site publishes as Markdown for agents comes back as the site wrote it — cheaper and truer than extracting it from HTML. It obeys robots.txt, never logs in, never solves CAPTCHAs, and refuses platforms whose terms forbid automated access. PDF and office-file URLs are answered with a pointer to pdf-to-markdown or document-to-markdown. Reading a page is the most common thing an agent does: this Actor is payable per request by an agent, priced per page, and returns only the content — no navigation, footers or cookie banners eating your context.

Call it from an agent

Through Apify's MCP server (https://mcp.apify.com), tool call-actor:

{"actor": "hermeskd/url-to-markdown", "input": {"urls": ["https://docs.apify.com/platform/actors/publishing/monetize"]}, "callOptions": {"maxTotalChargeUsd": 0.05, "timeout": 600}, "waitSecs": 30}

The call returns the run's status and its defaultDatasetId. Read the records with get-dataset-items:

{"datasetId": "<defaultDatasetId from the call>", "limit": 50}

As a dedicated tool, add https://mcp.apify.com?tools=hermeskd/url-to-markdown to your MCP client. A page finishes well inside the 30-second wait, so one call-actor plus one get-dataset-items is the whole exchange. maxTotalChargeUsd caps what the run can charge; when it is reached the remaining URLs come back as charge_limit_reached records and are not charged.

Input

Field	Type	Required	Default	Meaning
urls	array of strings	yes	—	Web page URLs, http or https, 1–100 per run.
render	enum auto · never · always	no	auto	auto renders a page in a browser only when its HTML yields under 200 words of main content; never uses the HTML only; always renders every page.
includeLinks	boolean	no	false	Add the page's links as {text, url}, absolute and deduplicated, at most 1,000.
maxChars	integer	no	200000	The Markdown is cut beyond this many characters (1,000–1,000,000).

Every limit above is enforced by the input schema and again in code.

Output

One record per input URL.

Field	Meaning
url	The input URL.
finalUrl	The page's URL after redirects.
ok	true when the page converted; false with error.
status	The final HTTP status; carried on http_error records too.
title	The page's title (its <title>, or its og:title, or its first heading); for a page served as Markdown, its front matter's title or its first heading.
description	The page's meta description, when it has one.
lang	The page's declared language, when it has one.
markdown	The main content as Markdown — headings, lists, tables and emphasis; navigation, footers and other boilerplate left out; cut at maxChars. A page the server sends as Markdown is returned as the site wrote it, its front matter moved into title, description and lang. A plain-text URL is returned as its text, and JSON, XML and CSV as a fenced block.
wordCount	Words in the Markdown returned.
rendered	true when a browser rendered the page (charged as page-rendered), false for the HTML alone (page-converted).
contentFormat	What the server sent and the Markdown was made from: html (main content extracted), markdown (the site's own Markdown), text, or json, xml or csv.
links	With includeLinks: the page's links as {text, url}.
error	{code, message} when ok is false.

Example record for the call above. The docs site answers this Actor with its own Markdown; the record is the one this release returns for the response the site sent on 2026-10-16 (status, final URL and body as served; this Actor's fixture test checks it field by field). The Markdown is shortened here:

{
  "url": "https://docs.apify.com/platform/actors/publishing/monetize",
  "finalUrl": "https://docs.apify.com/actors/publishing/monetize",
  "ok": true,
  "status": 200,
  "title": "Monetize Actors",
  "description": null,
  "lang": null,
  "markdown": "> ## Documentation index\n> Fetch the complete documentation index at: https://docs.apify.com/llms.txt\n> Use this file to discover all available pages before exploring further.\n\n# Monetize Actors\n\nYou can monetize your automation, AI agent, and web scraping projects by publishing them as paid Actors on [Apify Store](https://apify.com/store).\n\n## Pricing models\n\n…",
  "wordCount": 762,
  "rendered": false,
  "contentFormat": "markdown"
}

Events and prices

Pay per event; platform usage is included in the price, never passed through.

Event	Title	Charged when	FREE / BRONZE	SILVER	GOLD / PLATINUM / DIAMOND
page-converted	Page converted	One page fetched and converted without a browser.	$0.001	$0.0009	$0.0008
page-rendered	Page rendered and converted	One page rendered in a browser and converted; charged instead of page-converted.	$0.003	$0.0027	$0.0024

Apify also charges its standard apify-actor-start event: $0.00005 per GB of run memory (1 event, $0.00005, at the default 1,024 MB). The minimum maximum-charge a caller can set is $0.01.

Worked example on the FREE tier: 20 documentation pages, 18 read from their HTML and 2 that needed a browser, cost 18 × $0.001 + 2 × $0.003 = $0.024, plus $0.00005 for the start, $0.02405 in all. A page is charged once, as one event or the other; nothing is charged for a failed page. A page that converts but has no text is still charged — the fetch and conversion were done.

Limits and run time

• 1–100 URLs per run; pages up to 10 MB; the Markdown cut at maxChars (200,000 characters by default).
• Each request — its redirects and its whole body included — must finish within 30 seconds (the per-request timeout; a slower page is download_failed); at most 2 page loads a second go to one host, whether fetched or rendered. Up to four pages are worked on at once, and the records keep the input order.
• A rendered page is read when its document has loaded and its network has been quiet for half a second, and never more than 5 seconds after the document; one page renders at a time at the default memory (1,024 MB), two at 2,048 MB or more.
• A page the server sends as Markdown (by its content type, or by its body when the type is generic, such as text/plain) is never rendered, whatever its length, and is charged as page-converted. Every request asks for Markdown first: Accept: text/markdown, text/html;q=0.9, application/xhtml+xml;q=0.9, text/plain;q=0.8, */*;q=0.5.
• With render: always, a URL that turns out to be a file or plain text, not an HTML page, is answered as a plain fetch answers it and charged as page-converted (or not at all, for a file another Actor converts). With render: auto, a render that fails for any reason but robots.txt, the deny-list or a sign-in wall returns the page read from its HTML, charged as page-converted.
• noindex and other indexing directives govern search engines, not fetching: such a page is converted. robots.txt is what decides whether a page may be fetched.
• Typical run time: a static page in about a second, a rendered page in a few seconds, at the default memory (1,024 MB). The default run timeout is 600 seconds.

Errors

Errors are records with ok: false and error.code; none is charged.

error.code	Meaning
download_failed	The page could not be fetched (network error, timeout, or the browser could not load it).
http_error	The final HTTP status was not 2xx; status carries it.
unsupported_content_type	Not a web page — PDF, office, image, audio and video URLs name the Hermes Actor that converts them, under every render mode; legacy .doc, .xls and .ppt files are not read by any Hermes Actor.
too_large	Larger than 10 MB.
robots_disallowed	robots.txt forbids fetching the URL, or a page tried to navigate to a URL it forbids.
blocked_host	The host is on the deny-list, or resolves to a private or local address.
login_required	The page requires sign-in (HTTP 401, or a sign-in wall); nothing was attempted.
timeout	The page did not finish within the run's time budget.
internal_error	Unexpected failure; message carries the class.
charge_limit_reached	The run reached its maximum total charge before this URL; it was not processed.
invalid_input	The run input passed the input schema but cannot be used (for example no URL at all); the message names the field. One record per input given, or one with url: null when none was given; nothing is processed.

The run fails only when every input failed with an Actor-side code (internal_error, timeout), so a broken Actor is visible. Every other run succeeds — including one where nothing converted because of the caller's own inputs (bad URL, robots, deny-list, a sign-in wall, invalid input, the charge limit), or where those and Actor-side failures are mixed.

Sources and terms

Public pages the caller names. Process only pages you have the right to process. Fetching follows the Hermes fetch policy: robots.txt is obeyed for the user-agent HermesBot (and *), on every redirect and, in the browser, for every frame's document, and a disallowed URL is refused without a request; platforms whose terms forbid automated access — among them facebook.com, instagram.com, tiktok.com, youtube.com, linkedin.com, x.com, reddit.com, amazon, indeed and Google Search — and people-search sites are on a deny-list and refused without a request, also for anything a rendered page loads; private and local addresses are refused, after DNS and on every redirect; the user-agent is HermesBot/1.0 (+https://apify.com/hermeskd) (a rendering browser keeps its own string with HermesBot appended); no login, cookies, credentials, CAPTCHA solving, evasion or proxies are used. The page's author fields are never returned.

Not affiliated

This Actor is not affiliated with Apify's own page fetchers or with any site it reads. trafilatura, Beautiful Soup, lxml, Playwright and Chromium are used under their open-source licences.

Related Hermes Actors

• pdf-to-markdown — PDF documents to Markdown, with OCR for scanned pages.
• document-to-markdown — Word, PowerPoint, Excel and other office files to Markdown.
• image-to-text — OCR for images.
• audio-to-transcript — speech to text.
• video-frames — frames and metadata from a video.
• image-upscale — image upscaling.
• audio-video-convert — audio and video format conversion.

Changelog

• 1.0 (2026-10-16) — every request asks for Markdown first; a page the server sends as Markdown is returned as the site wrote it, its front matter in title, description and lang, never rendered; new field contentFormat.
• 1.0 (2026-10-16) — files under render: always are answered like a plain fetch instead of rendered as empty pages; any failed render under render: auto keeps the static page; legacy Office files get no pointer.
• 1.0 (2026-10-16) — first release: main content as Markdown, a browser only when needed, links on request, robots.txt for every frame.