Website to Markdown for RAG — documentation and knowledge-base extraction

Give it public documentation or article URLs. Get back readable page content as Markdown, with the navigation, cookie banners, ads, share buttons and footers removed — plus the title, author, publish date, site name and language as separate fields.

It is built for the thing most people actually want a website content crawler for: filling a RAG index, a vector database or an LLM prompt with readable text.

• $2 per 1,000 pages delivered. Flat. Not compute units, not a monthly rental.
• Pages it cannot read are free. A page that 403s, times out, or turns out to be an empty shell is returned as a row with the reason — and is not charged.
• 49/50 documentation URLs extracted (98%) in the 2026-09-26 test. A separate mixed-site test returned 43/50 (86%). These are different samples, not a universal success-rate claim.
• HTTP first, browser fallback when needed. All 49 successful documentation pages in that test used HTTP.

Try the ready-to-run documentation RAG example, or start with the smaller input below.

Try a two-page RAG extraction — at most $0.004

Click Try for free, open the JSON input editor, and paste:

{
  "startUrls": [
    {"url": "https://docs.apify.com/actors"},
    {"url": "https://blog.cloudflare.com/workers-ai/"}
  ],
  "maxCrawlDepth": 0,
  "maxPages": 2,
  "maxItems": 2,
  "includeImages": false,
  "outputFormat": "markdown"
}

Click Start, then open Output → All pages (JSON). Check success, markdown and finalUrl. Export JSON or CSV when the text looks right. The Actor charge is $0.002 per successfully extracted page, so this example delivers at most two paid pages. Apify account credits and eligibility are separate from this price; “Try for free” does not mean unlimited free extraction.

Recipe: refresh a documentation knowledge base

For a small documentation crawl, replace the URL and glob with your own documentation domain:

{
  "startUrls": [{"url": "https://docs.apify.com/actors"}],
  "maxCrawlDepth": 2,
  "sameDomainOnly": true,
  "includeUrlPatterns": ["https://docs.apify.com/actors/**"],
  "maxPages": 50,
  "maxItems": 50,
  "includeImages": false,
  "outputFormat": "markdown"
}

This caps delivery at 50 pages ($0.10). maxPages caps page visits, including failed pages; maxItems caps successful, billable pages. Save the input as a task and schedule it weekly if your source changes regularly. Each scheduled run is a fresh crawl and is charged again for successful pages. The Actor does not detect changes, create embeddings, or upload to your vector database.

In your indexing step, keep only success: true rows, split markdown into chunks, and retain finalUrl, title and scrapedAt as citation metadata. Upsert by source URL to replace old content. This Actor supplies readable source content; retrieval and answer generation belong in your application.

Use from Python or n8n

Install apify-client and set your own APIFY_TOKEN environment variable. This uses an asynchronous run and waits for completion rather than relying on a short HTTP response timeout:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("munin/url-to-markdown").call(run_input={
    "startUrls": [{"url": "https://docs.apify.com/actors"}],
    "maxPages": 1,
    "maxItems": 1,
    "includeImages": False,
})
if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Extraction run ended with {run['status']}")
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page.get("success"):
        print(page["finalUrl"], page["markdown"])
    else:
        print("Skipped:", page["url"], page.get("error"))

In n8n, use the Apify node to Run an Actor with the same JSON input, wait for completion, then Get dataset items using defaultDatasetId. Filter by success before sending markdown to your model or vector store. Store the API token in n8n credentials, not in a public workflow export.

What a row looks like

{
  "url": "https://blog.cloudflare.com/workers-ai/",
  "finalUrl": "https://blog.cloudflare.com/workers-ai/",
  "title": "Workers AI: serverless GPU-powered inference on Cloudflare's global network",
  "siteName": "The Cloudflare Blog",
  "author": "Rita Kozlov",
  "publishedAt": "2023-09-27T14:00:00.000Z",
  "language": "en",
  "wordCount": 1830,
  "markdown": "# Workers AI: serverless GPU-powered inference...\n\nToday...",
  "extractedBy": "http/readability",
  "httpStatus": 200,
  "success": true
}

extractedBy tells you which route produced the row — http, proxy or browser, and whether the body came from the article extractor, from the page's JSON-LD, or from a content container. You can audit any row without re-running it.

How it gets the content

Three routes, cheapest first. A page stops at the first route that yields real text.

1. Plain HTTP with a normal browser's headers. No proxy, so no proxy cost. Most pages end here.
2. Retry through a proxy, with a fresh IP — only for pages that answered 401, 402, 403, 406, 429, 451, 5xx, or dropped the connection. Changing IP cannot fix a 404, so a 404 never retries.
3. A real browser (Chromium), only for pages that returned HTML with no readable body — single-page apps that build their content in JavaScript. Images, fonts and video are blocked during the render, which roughly halves the time.

Content selection is Mozilla's Readability — the same extractor behind Firefox Reader View — not a homegrown heuristic. When Readability finds nothing, the actor falls back to the page's JSON-LD articleBody, then to <main> / <article>, in that order.

A page whose extracted body is shorter than "Minimum words" (default 25) produces an uncharged failure row. The extractor also rejects recognized security-challenge pages. These checks reduce empty or irrelevant output, but do not replace reviewing a sample before bulk ingestion. Failure rows are not charged.

Common uses

• Build a RAG corpus — point it at a documentation site with "Follow links, how deep" set to 2 and let it walk the docs.
• Feed a prompt — turn "Keep links" off so the text going into embeddings is not full of URLs.
• Watch a set of pages — schedule it on a fixed URL list and diff markdown between runs.
• Archive articles — keep markdown, title, author and publishedAt as a permanent record.

Crawling

Set Follow links, how deep above 0 and the actor also converts the pages your start URLs link to. It stays on the start URLs' domains unless you turn that off, and you can narrow it further with glob patterns:

Setting	Example	Effect
Only follow URLs matching	https://docs.example.com/guide/**	Only the guide section is crawled
Never follow URLs matching	**/tag/**	Tag archive pages are skipped
Max pages	500	The run stops after 500 pages, successes and failures together
Max pages to deliver	200	The run stops after 200 successful pages, so your cost is capped

Glob patterns apply to followed links only. Your start URLs are always fetched.

What it does not do

Written out so you can decide before you spend anything.

• It does not break bot protection. Sites behind an aggressive challenge stay unreadable even with residential proxy and a real browser. The mixed-site test could not extract AP News, Reuters, IMDb, Serious Eats, Allrecipes, NerdWallet and Investopedia. Failed pages return an error row and are not charged. Site behavior changes over time.
• It does not read PDFs, Word files, images or video. A non-HTML URL is returned as a failed row saying so. It does not retry them.
• It does not log in. No cookies, credentials or session input. Pages behind a login or a hard paywall are not reachable.
• It does not execute a site's search or fill forms. It follows <a href> links only.
• It does not render JavaScript by default. It renders only when plain HTTP gave nothing readable. Set "JavaScript rendering" to Always if you know a whole site needs it — slower and slightly more expensive, but not charged differently.
• It does not guarantee a publish date. publishedAt comes from the page's own metadata. Sites that do not publish one leave the field null; it is never guessed.
• Infinite scroll is not scrolled. Only the content present after load is captured.

Pricing

$2.00 per 1,000 pages delivered ($0.002 per page), charged per successfully extracted page.

Nothing else is charged: no monthly rental, no compute units, no per-run minimum. A page that fails, a page that is too short, and a page that turns out to be a PDF are all free. Use Max pages to deliver to put a hard ceiling on a run.

Benchmark — 2026-09-26

Two separate samples were run on Apify, with at most 50 pages per run:

Sample	Pages extracted	Failed pages	Notes
Documentation: Apify, Python, PostgreSQL	49 / 50 (98%)	1	A 20-word PostgreSQL chapter index was below the default 25-word minimum. Its feedback footer was not substituted for the chapter.
Mixed sites: documentation, news, blogs, forums and other pages	43 / 50 (86%)	7	Blocked pages and detected security challenges were returned as uncharged failures.

The documentation sample was selected before measurement: the first links in breadth-first document order, 17 from Apify, 17 from Python and 16 from PostgreSQL. It is a three-site sample, not evidence for every documentation platform. The mixed sample reused the earlier 50-URL list. Residential proxy was enabled for the mixed sample.

Success here means the extractor returned content above the configured minimum. It does not certify completeness or suitability for your RAG application. Inspect Markdown from your target site before scheduling a large crawl. A SUCCEEDED run can contain failed page rows.

Input in brief

Field	Default	What it is for
Start URLs	—	The pages to convert. One per line.
Follow links, how deep	0	0 = only your URLs. 1+ crawls onward.
Output format	Markdown	Markdown, plain text, cleaned HTML, or all three.
Keep links / Keep images	on	Turn off for cleaner embedding input.
Remove these elements	—	CSS selectors to delete before extraction.
Minimum words	25	Below this, the page counts as failed and is free.
JavaScript rendering	Auto	Auto / Never / Always.
Parallel pages	8	Raise for speed, lower to be gentle on a site.
Max pages to deliver	0	Caps what a run can cost.

Notes on responsible use

Crawl what you are allowed to crawl. This actor fetches pages the way a browser does and does not attempt to defeat access controls, log in, or hide what it is. Respect the terms of the sites you point it at, and keep personal data out of what you collect.