URL to Markdown for LLM & RAG: fast, cheap web page to Markdown converter

Convert any public web page into clean Markdown or plain text that LLMs, AI agents and RAG pipelines can use directly. The main content is extracted (menus, cookie banners, footers and ads removed), metadata is added (title, description, language, published date), and the text can be pre-chunked for embeddings. Optionally crawl a whole website or docs section by following same-site links.

It uses plain HTTP with no headless browser, so it is very fast (about 0.5 s per page) and costs a fraction of browser-based tools.

Use cases

• Feed web pages to ChatGPT, Claude, Gemini, LangChain, LlamaIndex or your own agent
• Build a RAG knowledge base from documentation sites, blogs and help centers
• Monitor articles and turn them into text for summarization or classification
• Website-to-Markdown export for content migration

Input

{
  "urls": ["https://docs.python.org/3/tutorial/index.html"],
  "extractMode": "main",
  "outputFormat": "markdown",
  "crawlDepth": 1,
  "maxPages": 50,
  "includeUrlRegex": ["/tutorial/"],
  "chunkSize": 2000,
  "chunkOverlap": 200
}

• extractMode: main (article or main content only, the default) or full (whole page as Markdown).
• crawlDepth: 0 converts only the listed URLs. 1 or more also follows links on the same website.
• includeUrlRegex: restrict the crawl to matching URLs (for example /docs/).

New options (v0.2)

Option	What it does
excludeUrlRegex	Skip crawled links matching a pattern
useSitemap	Seed the crawl from sitemap.xml (found via robots.txt)
removeSelectors / keepSelectors	Delete nav/footer/cookie banners, or convert only article, main, ...
includeImages	Keep images as Markdown image links
saveHtml	Raw HTML saved to the key-value store (htmlKey)
generateLlmsTxt	Writes llms.txt and llms-full.txt for the converted pages
maxResults	Hard stop after N converted pages (cost cap)
requestHeaders	Extra headers for your own sites (never logged)
onlyChanged, stateKey	Scheduled monitoring: unchanged pages are skipped for free

Not included on purpose: proxies, "stealth" browsers, CAPTCHA or login bypass. Pages that need JavaScript return no_text.

Output (one item per page)

{
  "url": "https://docs.python.org/3/tutorial/classes.html",
  "status": "ok", "statusCode": 200, "depth": 1,
  "title": "9. Classes", "description": "…", "language": "en", "publishedDate": null,
  "content": "# 9. Classes\n\nClasses provide a means of bundling data and functionality together…",
  "wordCount": 5420, "charCount": 34120,
  "chunks": [{"index": 0, "charStart": 0, "text": "…"}],
  "durationMs": 772
}

status is ok, no_text (the page only renders with JavaScript), skipped (robots.txt) or error.

Pricing (pay per event)

• $0.001 per converted page ($1 per 1,000 pages), plus Apify's default $0.00005 start fee per run. That is cheaper than Web Fetch ($1.50 per 1,000) and RAG Web Browser ($2.28 per 1,000).
• Example: a 300-page documentation site costs about $0.30.
• Errors, robots.txt-skipped pages and pages without text are free.

Performance (measured 2026-10-21)

Test	Result
Wikipedia "Large language model"	13,058 words, 52 chunks, 1.4 s
BBC News front page	1,298 words, 0.8 s
Python tutorial crawl, depth 1, 6 pages	6 pages in about 4 s total

Limits and good behavior

• No JavaScript rendering. Single-page apps that render everything client-side return no_text and are not charged.
• robots.txt is respected by default. No captcha or anti-bot bypass and no login.
• Only public http(s) URLs are fetched. Private and local network addresses are refused.
• For PDF, Word, PowerPoint and Excel files use our Document to Markdown Actor.
• You are responsible for having the right to use the content you convert.