URL to Markdown — Clean Text for LLM & RAG, With Selectors

URL to Markdown converts permitted public pages to clean Markdown and optional plain text, with HTML selectors, source links and provenance for LLM and RAG pipelines; rendering follows the selected mode and native Markdown stays unrendered. AI agents can call and pay through Apify MCP.

The active Apify pricing record determines your bill; for the previous/requested schedules, page-converted previous $0.001 / requested $0.0015; page-rendered previous/requested $0.003; usage is included and failed results are free.

Example result

Selected fields of a real result for the shown input below, fetched from the public Python documentation in this Actor's production image on 2026-10-28 at 1,024 MB and a quarter CPU. Markdown and text are shortened here and the links array is omitted; lengths, hash and truncation flags describe the full returned record, not this display excerpt. No platform charge or settlement is claimed for this local image run.

{
  "url": "https://docs.python.org/3/library/urllib.parse.html",
  "finalUrl": "https://docs.python.org/3/library/urllib.parse.html",
  "ok": true,
  "status": 200,
  "title": "urllib.parse — Parse URLs into components — Python 3.15.2 documentation",
  "description": "Source code: Lib/urllib/parse.py This module defines a standard interface to break Uniform Resource Locator (URL) strings up in components (addressing scheme, network location, path etc.), to combi...",
  "lang": "en",
  "markdown": "# `urllib.parse` — Parse URLs into components\n\n…",
  "wordCount": 4577,
  "rendered": false,
  "contentFormat": "html",
  "text": "urllib.parse\n— Parse URLs into components\nSource code:\nLib/urllib/parse.py\nThis module defines a standard interface to break Uniform Resource Locator (URL)\nstrings up in components (addressing scheme, network location, path etc.), to\ncombine the components back into a URL string, and to convert a “r…",
  "textTruncated": false,
  "canonicalUrl": "https://docs.python.org/3/library/urllib.parse.html",
  "fetchedAt": "2026-10-28T06:05:22.805534Z",
  "contentSha256": "6871c294e9d8ecc4ce538d1a0ab09f75c327365df83119ffa4e7febcbe357968",
  "originalChars": 41236,
  "returnedChars": 41236,
  "truncated": false
}

Python documentation: Copyright (c) Python Software Foundation, PSF License Version 2. The example converts HTML to Markdown/text, removes selected boilerplate and resolves source links; the source URL is retained. Source notices are distributed with this Actor.

How to use it

Start a run in Apify Console, by API or from an agent with an input like this one — it is the input Apify runs every day to test this Actor:

{"urls": ["https://docs.python.org/3/library/urllib.parse.html"], "render": "never", "includeSelector": ".body", "removeSelectors": [".headerlink", ".viewcode-link"], "includeLinks": true, "includeText": true, "preserveContentLinks": true, "includeProvenance": true}

Start with the minimal JSON example above, then adjust the fields in the table below. Read the Example result and Output fields sections for the returned record, Errors for failure codes, and Events and prices for result events and the separate platform start.

Field	Type	Required	Default	Meaning
urls	array of strings	yes	—	Web page URLs, http or https, 1–100 per run.
render	enum auto · never · always	no	auto	auto renders a page in a browser only when its HTML yields under 200 words of main content; never uses the HTML only; always renders every page.
includeLinks	boolean	no	false	Add the page's links as {text, url}, absolute and deduplicated, at most 1,000.
maxChars	integer	no	200000	Markdown and optional text are each cut beyond this many Unicode characters (1,000–1,000,000).
includeSelector	string	no	—	CSS selector for the HTML elements to keep; 1–1,000 characters. Nested matches are not duplicated. HTML only; no readable match is an uncharged invalid_input.
removeSelectors	array of strings	no	[]	Up to 10 CSS selectors of 1–1,000 characters, removed only from the extraction copy. HTML only; no clicking or navigation.
includeText	boolean	no	false	Add text and textTruncated. The text view is separately capped at maxChars; Markdown remains available.
preserveContentLinks	boolean	no	false	Keep HTTP(S) links within extracted HTML Markdown and resolve them against finalUrl. Native Markdown is never rewritten.
includeProvenance	boolean	no	false	Add declared canonical URL, fetch time, full converted Markdown SHA-256 and truncation lengths/flag; no author or contact metadata.

Every limit above is enforced by the input schema and again in code. Options are off when omitted; the shown daily-test input explicitly enables them. includeSelector and removeSelectors require HTML; using them with a native Markdown, plain text, JSON, XML or CSV response returns an uncharged invalid_input, rather than silently ignoring them. With render: auto, a missing HTML scope can be retried after rendering. Selection never bypasses a sign-in wall or robots refusal.

For already-served Markdown, includeText removes common heading/list/quote/table/link/emphasis syntax and keeps fenced code literal; it is a readable text view, not an HTML renderer. The original Markdown and its source citations remain in markdown. HTML selectors operate on an extraction copy; title, description, language and declared canonical metadata come from the original document.

Events and prices

The active Apify pricing record determines your bill; each schedule below is conditional on that exact record being active. A pending record does not authorise its charges. Before submission, while pending, and after activation, use the actual active record, not the requested calendar, to determine the bill.

Requested activation: 2026-12-06T06:00:00Z. This is a request until confirmed by the record; the record supplies the actual effective instant.

Previous schedule — only if that record is active

apify-actor-start: FREE $0.00005, BRONZE $0.00005, SILVER $0.00005, GOLD $0.00005, PLATINUM $0.00005, DIAMOND $0.00005; page-converted: FREE $0.001, BRONZE $0.001, SILVER $0.0009, GOLD $0.0008, PLATINUM $0.0008, DIAMOND $0.0008; page-rendered: FREE $0.003, BRONZE $0.003, SILVER $0.0027, GOLD $0.0024, PLATINUM $0.0024, DIAMOND $0.0024.

Requested schedule — only if that record is active

Event	Title	Charged when	FREE / BRONZE	SILVER	GOLD / PLATINUM / DIAMOND
page-converted	Page converted	One page fetched and converted without a browser.	$0.0015	$0.00135	$0.0012
page-rendered	Page rendered and converted	One page rendered in a browser and converted; charged instead of page-converted.	$0.003	$0.0027	$0.0024

Platform event in the requested schedule: apify-actor-start: FREE $0.00005, BRONZE $0.00005, SILVER $0.00005, GOLD $0.00005, PLATINUM $0.00005, DIAMOND $0.00005.

The requested start at default memory is (1 event, $0.00005, at the default 1,024 MB).

Worked event subtotal, requested schedule at FREE: if 10 page-converted events are charged, 10 × $0.0015 = $0.015 for that event alone. All other charged events and the platform start event add their listed fees; this subtotal is not a run total. The configured maximum-charge floor is $0.01. Platform usage is included; failed results are uncharged.

Call it from an agent

Through Apify's MCP server (https://mcp.apify.com), tool call-actor:

{"actor": "hermeskd/url-to-markdown", "input": {"urls": ["https://docs.python.org/3/library/urllib.parse.html"], "render": "never", "includeSelector": ".body", "removeSelectors": [".headerlink", ".viewcode-link"], "includeLinks": true, "includeText": true, "preserveContentLinks": true, "includeProvenance": true}, "callOptions": {"maxTotalChargeUsd": 0.05, "timeout": 600}, "waitSecs": 30}

The call returns the run's status and its defaultDatasetId. Read the records with get-dataset-items:

{"datasetId": "<defaultDatasetId from the call>", "limit": 50}

As a dedicated tool, add https://mcp.apify.com?tools=hermeskd/url-to-markdown to your MCP client. A page finishes well inside the 30-second wait, so one call-actor plus one get-dataset-items is the whole exchange. maxTotalChargeUsd caps what the run can charge; when it is reached the remaining URLs come back as charge_limit_reached records and are not charged.

Output fields

One record per input URL.

Field	Meaning
url	The input URL.
finalUrl	The page's URL after redirects.
ok	true when the page converted; false with error.
status	The final HTTP status; carried on http_error records too.
title	The page's title (its <title>, or its og:title, or its first heading); for a page served as Markdown, its front matter's title or its first heading.
description	The page's meta description, when it has one.
lang	The page's declared language, when it has one.
markdown	The main content as Markdown — headings, lists, tables and emphasis; navigation, footers and other boilerplate left out; cut at maxChars. A page the server sends as Markdown is returned as the site wrote it, its front matter moved into title, description and lang. A plain-text URL is returned as its text, application/json and application/xml as a fenced block. Existing text/xml and text/csv responses remain raw text (contentFormat: text), including with optional outputs.
wordCount	Words in the Markdown returned.
rendered	true when a browser rendered the page (charged as page-rendered), false for the HTML alone (page-converted).
contentFormat	What the server sent and the Markdown was made from: html (main content extracted), markdown (the site's own Markdown), text, or json, xml or csv.
links	With includeLinks: the page's links as {text, url}.
text	With includeText: plain text of extracted HTML, a plain view of common Markdown syntax for served Markdown, or the literal plain-text/code body. Markdown is always retained.
textTruncated	With includeText: whether maxChars shortened the text view.
canonicalUrl	With includeProvenance: the HTML document's declared HTTP(S) canonical link, resolved against finalUrl, or null. Never fetched or substituted for the final URL.
fetchedAt	With includeProvenance: UTC ISO timestamp after the fetch completes (or after successful browser extraction).
contentSha256	With includeProvenance: SHA-256 of UTF-8 full converted Markdown before truncation; not the raw response or the shortened output.
originalChars	With includeProvenance: Unicode characters in full converted Markdown before maxChars.
returnedChars	With includeProvenance: Unicode characters in the returned markdown field.
truncated	With includeProvenance: whether maxChars shortened Markdown; its hash still describes the full converted content.
error	{code, message} when ok is false.

Limits and run time

• 1–100 URLs per run; pages up to 10 MB; the Markdown cut at maxChars (200,000 characters by default).
• Each request — its redirects and its whole body included — must finish within 30 seconds (the per-request timeout; a slower page is download_failed); at most 2 page loads a second go to one host, whether fetched or rendered. Up to four pages are worked on at once, and the records keep the input order.
• A rendered page is read when its document has loaded and its network has been quiet for half a second, and never more than 5 seconds after the document; one page renders at a time at the default memory (1,024 MB), two at 2,048 MB or more.
• A page the server sends as Markdown (by its content type, or by its body when the type is generic, such as text/plain) is never rendered, whatever its length, and is charged as page-converted. Every request asks for Markdown first: Accept: text/markdown, text/html;q=0.9, application/xhtml+xml;q=0.9, text/plain;q=0.8, */*;q=0.5.
• With render: always, a URL that turns out to be a file or plain text, not an HTML page, is answered as a plain fetch answers it and charged as page-converted (or not at all, for a file another Actor converts). With render: auto, a render that fails for any reason but robots.txt, the deny-list or a sign-in wall returns the page read from its HTML, charged as page-converted.
• noindex and other indexing directives govern search engines, not fetching: such a page is converted. robots.txt is what decides whether a page may be fetched.
• Typical run time: a small static page can spend about a second on work; the feature-enabled HTML smoke input took about 6–7 seconds including local startup at a quarter CPU, before platform image-pull time; rendered pages take longer, at the default memory (1,024 MB). The default run timeout is 600 seconds.

Errors

Errors are records with ok: false and error.code; none is charged.

error.code	Meaning
download_failed	The page could not be fetched (network error, timeout, or the browser could not load it).
http_error	The final HTTP status was not 2xx; status carries it.
unsupported_content_type	Not a web page — PDF, office, image, audio and video URLs name the Hermes Actor that converts them, under every render mode; legacy .doc, .xls and .ppt files are not read by any Hermes Actor.
too_large	Larger than 10 MB.
robots_disallowed	robots.txt forbids fetching the URL, or a page tried to navigate to a URL it forbids.
blocked_host	The host is on the deny-list, or resolves to a private or local address.
login_required	The page requires sign-in (HTTP 401, or a sign-in wall); nothing was attempted.
timeout	The page did not finish within the run's time budget.
internal_error	Unexpected failure; message carries the class.
charge_limit_reached	The run reached its maximum total charge before this URL; it was not processed.
invalid_input	The run input passed the input schema but cannot be used (for example no URL at all); the message names the field. Malformed options are refused before fetching; a missing/empty HTML selection or a selector on another format gives one uncharged record for that URL. One record per input given, or one with url: null when none was given.

The run fails only when every input failed with an Actor-side code (internal_error, timeout), so a broken Actor is visible. Every other run succeeds — including one where nothing converted because of the caller's own inputs (bad URL, robots, deny-list, a sign-in wall, invalid input, the charge limit), or where those and Actor-side failures are mixed.

Sources and terms

Public pages the caller names. Process only pages you have the right to process. Fetching follows the Hermes fetch policy: robots.txt is obeyed for the user-agent HermesBot (and *), on every redirect and, in the browser, for every frame's document, and a disallowed URL is refused without a request; platforms whose terms forbid automated access — among them facebook.com, instagram.com, tiktok.com, youtube.com, linkedin.com, x.com, reddit.com, amazon, indeed and Google Search — and people-search sites are on a deny-list and refused without a request, also for anything a rendered page loads; private and local addresses are refused, after DNS and on every redirect; the user-agent is HermesBot/1.0 (+https://apify.com/hermeskd) (a rendering browser keeps its own string with HermesBot appended); no login, cookies, credentials, CAPTCHA solving, evasion or proxies are used. The page's author fields are never returned.

Not affiliated

This Actor is not affiliated with Apify's own page fetchers or with any site it reads. trafilatura and Playwright (Apache-2.0), Beautiful Soup, Soup Sieve and markdownify (MIT), lxml (BSD-3-Clause) and Chromium are used under their open-source licences. Dependency copyright notices remain in their installed distributions; no model or external search data is added.

Related Hermes Actors

• AI Crawler Checker: Which AI Bots robots.txt Allows per URL — check declared robots.txt permissions for six AI crawlers at URL paths.
• Article Extractor — News & Blog Text, Date and Markdown by URL — an article's clean text, title and publish date, without the rest of the page.
• ATS Job Search — Jobs Confirmed Open on Company Career Boards — search open jobs in the Hermes directory of Greenhouse and Lever boards, each confirmed live.
• Speech to Text — Transcribe Audio & Video URLs, Speaker Labels — speech to text with SRT and VTT subtitles.
• Video Converter — MP4 to MP3, Trim, GIF and Audio Extraction — audio and video conversion: MP4 to MP3, trim, GIF.
• Broken Links Checker — Links, Assets, Redirects and Their Pages — broken links, 404s and redirects on a site.
• ATS Detector — Company Hiring System & Job Board Lookup — which hiring system and job board a company uses.
• SEC EDGAR Filings — Recent Company Forms, Dates and XBRL Flags — Read recent company financial filing metadata from the official SEC JSON API
• DOI Lookup — Crossref Titles, Venues, Dates and Citation Counts — crossref doi metadata.
• CVE Vulnerability Lookup — NVD CVSS Scores and Exploited Flags — look up NVD CVE dates, status, numeric CVSS scores and severity.
• Merge JSON Datasets — Deduplicate Rows, Export JSON and CSV — json dataset cleaner.
• DNS Lookup — Records, SPF Lookup Counts, DKIM & DMARC by Domain — DNS records and SPF, DKIM and DMARC checks.
• DOCX to Markdown — Word, Excel, PowerPoint and EPUB Converter — Word, Excel, PowerPoint and other office files to Markdown.
• Domain Availability Checker — Bulk RDAP Status and Registrar — whether domain names are available, from registry RDAP.
• USGS Earthquake Lookup — Event Magnitude, Depth, Coordinates — Read numeric USGS earthquake metadata for known event identifiers.
• Currency Exchange Rates — ECB Daily Euro Rates and Cross Pairs — Read latest ECB daily currency reference rates and calculated cross pairs.
• Bulk File Downloader — Save URL Lists to Storage With SHA-256 — files and images from URLs, saved with a public link each.
• GitHub Repo Stats — Stars, Forks, Subscribers and Open Issues — Read numeric metrics for known public GitHub repositories.
• Greenhouse Jobs — Live Openings, Offices and Apply Links — every open job on a company's Greenhouse board.
• Hacker News Story Metrics — Scores, Comment Counts, Timestamps — read numeric Hacker News story scores, timestamps and comment counts by ID.
• HTTP GET Request — Bulk Response Bodies, Status and SHA-256 — public http get.
• Security Headers Checker — HSTS, CSP, Framing, Nosniff by URL — Check seven declared HTTP security header families without returning raw values.
• Image Background Remover — U2NetP Cutouts for JPEG, PNG, WebP — Remove image backgrounds with the small U2NetP model and return transparent PNG pixels as base64
• Image Converter — JPEG, PNG and WebP With Resize and Quality — image format converter.
• Image Metadata Remover — EXIF and GPS Scrub, Pixels Verified — GPS location, camera serial numbers and other private metadata removed from photos, losslessly.
• Image to Text OCR — Text With Word Positions and Confidence — OCR: the text in images.
• Image Upscaler — Real-ESRGAN 2x/4x for Photos and Graphics — image upscaling 2x or 4x, and photo restoration.
• Lever Jobs Scraper — Job Board Postings With Salary, US & EU — every published job on a company's Lever site, global or EU.
• Lighthouse Audit — Lab Web Vitals and Scores, Mobile & Desktop — Lighthouse scores and Core Web Vitals.
• npm Package Lookup — Latest Version, License, Dependency Count — read latest npm package versions, license tokens and dependency counts.
• Website Change Monitor — Page Text Diffs Against Your Snapshots — what changed on a page since the last check.
• PDF to Markdown — PDF Parser & Text Extractor With OCR — PDF to Markdown and text, with OCR for scanned pages.
• Public Holidays Calendar — Holiday Dates for Five Countries — build offline public-holiday calendars for five countries and bounded years.
• Remote Jobs Search — Remote OK Jobs by Keyword, Tag and Salary — current remote jobs from Remote OK, with salary and listing links.
• Research Paper Search — OpenAlex Papers, DOIs, Citation Counts — Search OpenAlex for work-level paper metadata without authors, affiliations, abstracts or full text
• RSS Feed Reader — Atom and JSON Feed Parser With Enclosures — every item of an RSS, Atom or JSON feed.
• SEO Audit — On-Page Score, Meta Tags, Schema & Fix Guidance — an on-page SEO audit: meta tags, schema, headings and links.
• Sitemap Extractor — Unique URLs From XML, Gzip & Index Sitemaps — the URLs in a site's sitemaps.
• Text to Speech — Offline English, German, Spanish WAV Voices — Convert short nonpersonal texts to offline formant WAV speech for agents
• Time Zone Converter API — Convert one offset-qualified instant to named IANA time zones using pinned data.
• Extract Video Frames — Timed Stills by Interval or Scene Cut — frames, thumbnails and scenes from a video.
• Website Color Palette — CSS Colors With Alpha and Usage Counts — Extract declared website CSS color literals and counts from public HTML and bounded linked stylesheets
• Favicon Extractor — Icons, Touch Icons, Manifests, Theme Color — declared website icons, theme colors and manifest links.
• Website Screenshot — Full Pages, Elements, Dark Mode & PDF — screenshots and PDF prints of web pages.
• Tech Stack Detector — Website Technologies With Match Evidence — the technologies a website runs, detected live.
• Website Crawler to Markdown — Same-Site Pages for LLM and RAG — bounded whole-site Markdown with robots and crawl-delay enforced

Changelog

• 1.0 (2026-10-28 prepared release) — optional extraction selectors, plain text, inline source links and provenance; pricing changes are represented by the checked conditional schedules in Events and prices, not an assumed activation date. Store title no longer carries a price; text/xml and text/csv retain their existing raw-text records, including when optional outputs are requested.

• 1.0 (2026-10-26) — store listing rewritten for the people and agents searching for this job: title, description, search-engine title and description, categories, and the README in the order a searcher reads it (what it does, a real result, how to use it, prices, the agent call).

• 1.0 (2026-10-16) — every request asks for Markdown first; a page the server sends as Markdown is returned as the site wrote it, its front matter in title, description and lang, never rendered; new field contentFormat.

• 1.0 (2026-10-16) — files under render: always are answered like a plain fetch instead of rendered as empty pages; any failed render under render: auto keeps the static page; legacy Office files get no pointer.

• 1.0 (2026-10-16) — first release: main content as Markdown, a browser only when needed, links on request, robots.txt for every frame.