URL to Markdown for AI Agents and RAG

Give it a list of web page URLs. Get back each page's main content as clean Markdown, ready to paste into an LLM prompt, a vector database or a RAG pipeline. Navigation, cookie bars, scripts and styling are stripped out.

It reads pages over plain HTTP, the way a search engine bot does. There is no headless browser and no proxy in the loop, so it is fast (30 Wikipedia articles in about 10 seconds in our tests) and cheap: $0.40 per 1,000 pages, and pages that fail cost nothing.

It is a polite crawler: it reads each site's robots.txt and skips any page the site disallows, following RFC 9309, the internet standard for robots.txt.

What you get

One result per URL you send (a URL listed twice is read and charged once):

• markdown: the main content of the page as Markdown, with headings, lists, tables, code blocks and links (all links made absolute). It starts with the page title as a heading.
• title, description, language (as the page declares it, such as en or fr-CA) and publishedAt when the page states a publish date
• wordCount and tokenEstimate, so you can budget LLM context before you send anything
• links: up to 200 unique absolute links found on the whole page, handy for deciding what to read next
• needsJavaScript: true when the page shows little or no text without a browser (single-page apps)
• status and error: ok, or a plain-language reason the page was not converted
• url, finalUrl (after redirects), statusCode, contentType, fetchedAt

Input

Field	Required	Example	Notes
urls	yes	["https://en.wikipedia.org/wiki/Markdown"]	Full addresses, up to 10,000 per run. One result per unique URL

{
    "urls": [
        "https://en.wikipedia.org/wiki/Markdown",
        "https://www.rfc-editor.org/rfc/rfc9309.html"
    ]
}

Plain text pages (text/plain, text/markdown) come back as they are. PDFs, images and other files are skipped with status unsupported-content-type.

Output example

A real result, with the Markdown and links shortened:

{
    "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
    "finalUrl": "https://www.rfc-editor.org/rfc/rfc9309.html",
    "status": "ok",
    "statusCode": 200,
    "error": null,
    "contentType": "text/html",
    "title": "RFC 9309: Robots Exclusion Protocol",
    "description": "This document specifies and extends the \"Robots Exclusion Protocol\" method originally defined by Martijn Koster in 1994 ...",
    "language": "en",
    "publishedAt": null,
    "markdown": "# RFC 9309: Robots Exclusion Protocol\n\n| RFC 9309 | Robots Exclusion Protocol (REP) | September 2022 |\n| --- | --- | --- |\n...\n\n## Abstract\n\nThis document specifies and extends the \"Robots Exclusion Protocol\" method ...",
    "wordCount": 2940,
    "tokenEstimate": 5998,
    "links": ["https://www.rfc-editor.org/rfc/rfc9309", "https://www.rfc-editor.org/info/rfc9309", "..."],
    "needsJavaScript": false,
    "fetchedAt": "2026-10-31T16:05:12.016Z"
}

A page the site's robots.txt disallows still gets a row, so you always know what happened to every URL:

{
    "url": "https://en.wikipedia.org/wiki/Special:Random",
    "status": "blocked-by-robots",
    "error": "https://en.wikipedia.org/robots.txt disallows this page for crawlers, so it was skipped.",
    "markdown": null,
    "wordCount": 0
}

status	Meaning	Charged
ok	Converted	yes
needs-javascript	Little or no text without a browser	no
empty	The page has no readable text	no
blocked-by-robots	The site's robots.txt disallows it, or robots.txt had a server error	no
http-error	The site answered 4xx or 5xx (for example 403 or 404)	no
unsupported-content-type	Not HTML or plain text (PDF, image, ...)	no
too-many-redirects	More than 5 redirects	no
fetch-failed	No answer (timeout, unknown host)	no
invalid-url	Not a public http or https address	no

Pricing

Pay per event:

Event	Price	When
Page converted	$0.0004	Every result with status ok
Actor start	$0.00005	Once per run

Example: a run with 1,000 URLs where 950 pages convert costs 950 x $0.0004 + $0.00005 = $0.38005.

The actor checks your run's maximum charge before it fetches anything and only reads as many URLs as that covers, so a run never goes over the limit you set.

Limits

• Pages are read without running JavaScript. Sites that build their content in the browser come back with needsJavaScript: true and are not charged. Use a browser-based actor for those.
• Sites that block bots (for example with a 403 page or a challenge wall) return http-error. There is no proxy rotation or unblocking.
• Pages are read up to 4 MB of HTML. Very large pages (over 5,000 HTML elements) use a faster extraction: the page's main area with menus, sidebars and footers removed.
• tokenEstimate is an estimate, not a tokenizer: within about 6% of GPT-4o's tokenizer on English pages in our tests, rougher for other scripts.
• It reads the URLs you send. It does not crawl a whole site or run a web search.
• Up to 8 pages are fetched at a time. A URL listed more than once is read and charged once.

FAQ

How is this different from a full web crawler or a browser-based scraper? It does one job: URL in, clean Markdown out. No browser means it starts fast and costs a fraction of browser-based tools. The trade-off is that pages that only render with JavaScript are flagged instead of converted.

Does it respect robots.txt? Yes, always, and it cannot be turned off. It identifies itself as UrlToMarkdownBot with a link to this page, reads each site's robots.txt once per run, and follows RFC 9309: a missing robots.txt means everything is allowed, and a robots.txt that answers with a server error means the site is skipped.

Why did a URL come back with status http-error 403? The site refused automated requests. Nothing was charged for it.

Can I call it from an AI agent, MCP client or the API? Yes. Input is a single list of URLs, every field in the output has a fixed type, and it runs with limited permissions, so it needs no access to your other Apify data. It works from the Apify API, schedules, integrations and MCP clients.

Is there personal data in the results? The actor returns what is on the pages you choose. It targets no site by default and adds nothing of its own. You are responsible for having the right to use the pages you send it.

Something broke or a page converted badly. What do I do? Open an issue on the Issues tab with your run link and the URL. Issues are answered within a few days.