URL to Markdown — website content extractor for LLMs and RAG

Give it URLs. Get back the main content of each page as clean Markdown, with the navigation, cookie banners, ads, share buttons and footers removed — plus the title, author, publish date, site name and language as separate fields.

It is built for the thing most people actually want a website content crawler for: filling a RAG index, a vector database or an LLM prompt with readable text.

• $2 per 1,000 pages delivered. Flat. Not compute units, not a monthly rental.
• Pages it cannot read are free. A page that 403s, times out, or turns out to be an empty shell is returned as a row with the reason — and is not charged.
• 92% of pages extracted on a 50-site benchmark spanning docs, news, blogs, forums, recipes, government and Japanese sites. The 4 misses are listed below by name.
• Fast, because it does not open a browser unless it has to. 40 of those 50 pages were plain HTTP. A browser was needed for 5.

What a row looks like

{
  "url": "https://blog.cloudflare.com/workers-ai/",
  "finalUrl": "https://blog.cloudflare.com/workers-ai/",
  "title": "Workers AI: serverless GPU-powered inference on Cloudflare's global network",
  "siteName": "The Cloudflare Blog",
  "author": "Rita Kozlov",
  "publishedAt": "2023-09-27T14:00:00.000Z",
  "language": "en",
  "wordCount": 1830,
  "markdown": "# Workers AI: serverless GPU-powered inference...\n\nToday...",
  "extractedBy": "http/readability",
  "httpStatus": 200,
  "success": true
}

extractedBy tells you which route produced the row — http, proxy or browser, and whether the body came from the article extractor, from the page's JSON-LD, or from a content container. You can audit any row without re-running it.

How it gets the content

Three routes, cheapest first. A page stops at the first route that yields real text.

1. Plain HTTP with a normal browser's headers. No proxy, so no proxy cost. Most pages end here.
2. Retry through a proxy, with a fresh IP — only for pages that answered 401, 402, 403, 406, 429, 451, 5xx, or dropped the connection. Changing IP cannot fix a 404, so a 404 never retries.
3. A real browser (Chromium), only for pages that returned HTML with no readable body — single-page apps that build their content in JavaScript. Images, fonts and video are blocked during the render, which roughly halves the time.

Content selection is Mozilla's Readability — the same extractor behind Firefox Reader View — not a homegrown heuristic. When Readability finds nothing, the actor falls back to the page's JSON-LD articleBody, then to <main> / <article>, in that order.

A page whose extracted body is shorter than "Minimum words" (default 25) counts as a failure, not as a row. This is deliberate: returning a Markdown file containing nothing but a nav menu is the single most common complaint about extractors in this category. You are not charged for those.

Common uses

• Build a RAG corpus — point it at a documentation site with "Follow links, how deep" set to 2 and let it walk the docs.
• Feed a prompt — turn "Keep links" off so the text going into embeddings is not full of URLs.
• Watch a set of pages — schedule it on a fixed URL list and diff markdown between runs.
• Archive articles — keep markdown, title, author and publishedAt as a permanent record.

Crawling

Set Follow links, how deep above 0 and the actor also converts the pages your start URLs link to. It stays on the start URLs' domains unless you turn that off, and you can narrow it further with glob patterns:

Setting	Example	Effect
Only follow URLs matching	https://docs.example.com/guide/**	Only the guide section is crawled
Never follow URLs matching	**/tag/**	Tag archive pages are skipped
Max pages	500	The run stops after 500 pages, successes and failures together
Max pages to deliver	200	The run stops after 200 successful pages, so your cost is capped

Glob patterns apply to followed links only. Your start URLs are always fetched.

What it does not do

Written out so you can decide before you spend anything.

• It does not break bot protection. Sites behind an aggressive challenge stay unreadable even with residential proxy and a real browser. On the benchmark these were Reuters, Investopedia, Serious Eats and IMDb — 4 of 50. They come back as failed rows with the status code, and cost nothing.
• It does not read PDFs, Word files, images or video. A non-HTML URL is returned as a failed row saying so. It does not retry them.
• It does not log in. No cookies, credentials or session input. Pages behind a login or a hard paywall are not reachable.
• It does not execute a site's search or fill forms. It follows <a href> links only.
• It does not render JavaScript by default. It renders only when plain HTTP gave nothing readable. Set "JavaScript rendering" to Always if you know a whole site needs it — slower and slightly more expensive, but not charged differently.
• It does not guarantee a publish date. publishedAt comes from the page's own metadata. Sites that do not publish one leave the field null; it is never guessed.
• Infinite scroll is not scrolled. Only the content present after load is captured.

Pricing

$2.00 per 1,000 pages delivered ($0.002 per page), charged per successfully extracted page.

Nothing else is charged: no monthly rental, no compute units, no per-run minimum. A page that fails, a page that is too short, and a page that turns out to be a PDF are all free. Use Max pages to deliver to put a hard ceiling on a run.

Benchmark

50 real URLs, run on the Apify platform, 8 in parallel, residential proxy enabled, JavaScript rendering on Auto.

Pages extracted	46 / 50 = 92%
Route used	40 plain HTTP, 1 proxy retry, 5 browser render
Wall clock	32 seconds for all 50
Median body	711 words
Failures	4, all bot protection, all uncharged

The URL list is in test/urls.json in the actor's source, so the number is reproducible rather than a claim.

Input in brief

Field	Default	What it is for
Start URLs	—	The pages to convert. One per line.
Follow links, how deep	0	0 = only your URLs. 1+ crawls onward.
Output format	Markdown	Markdown, plain text, cleaned HTML, or all three.
Keep links / Keep images	on	Turn off for cleaner embedding input.
Remove these elements	—	CSS selectors to delete before extraction.
Minimum words	25	Below this, the page counts as failed and is free.
JavaScript rendering	Auto	Auto / Never / Always.
Parallel pages	8	Raise for speed, lower to be gentle on a site.
Max pages to deliver	0	Caps what a run can cost.

Notes on responsible use

Crawl what you are allowed to crawl. This actor fetches pages the way a browser does and does not attempt to defeat access controls, log in, or hide what it is. Respect the terms of the sites you point it at, and keep personal data out of what you collect.