Website to Markdown for LLM & RAG: Fast Crawler, No Browser

Turn any public website into clean, LLM-ready Markdown in seconds. Give it one page or a whole docs site; get back the main content (no menus, cookie banners or footers) with headings, lists, tables and code blocks with language tags preserved, plus title, description, author, date, language, a token estimate, a content hash and optional ready-made RAG chunks.

It uses fast HTTP requests instead of a headless browser, so it is quick and cheap: about 500 pages in under a minute in our tests.

You only pay for pages that were actually converted. 404s, timeouts, anti-bot pages, empty pages, PDFs/images and robots.txt-blocked URLs are free.

What you can use it for

• Feeding documentation, help centers, blogs and knowledge bases into a vector database / RAG pipeline (LangChain, LlamaIndex, Pinecone, OpenAI Assistants, custom GPTs)
• Giving AI agents a clean Markdown view of any web page (works via the Apify MCP server)
• Building training or fine-tuning datasets from public sites
• Content audits and change detection (compare contentHash between runs)

Features

• Main-content extraction (Mozilla Readability, with a sanity check against <article>/<main>), or whole-page mode
• GitHub-flavored Markdown: tables, strikethrough, fenced code with language
• Absolute links and image URLs; options to drop images or links to save tokens
• Crawling: depth limit, page limit, scope (same path / hostname / domain), include/exclude regex, optional sitemap discovery (robots.txt, sitemap indexes, .gz)
• RAG chunking on heading/paragraph boundaries with overlap (no extra charge)
• Polite by default: respects robots.txt and Crawl-delay, per-host delay, retries with backoff and Retry-After
• Fair crawling across multiple start URLs (one big site can't starve the others)
• Handles redirects (incl. broken https→http redirects), charsets, <base href>, canonical URLs, text/markdown responses
• Refuses private/internal addresses
• Stops cleanly at your max total charge without storing unpaid results

Input example

{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "maxCrawlDepth": 2,
  "maxPages": 200,
  "crawlScope": "samePath",
  "chunkSize": 2000
}

Output example (one row per page)

{
  "url": "https://docs.python.org/3/tutorial/appetite.html",
  "loadedUrl": "https://docs.python.org/3/tutorial/appetite.html",
  "statusCode": 200,
  "depth": 1,
  "title": "1. Whetting Your Appetite",
  "language": "en",
  "extractionMode": "article",
  "markdown": "# 1. Whetting Your Appetite\n\nIf you do much work on computers, eventually you find that there's some task you'd like to automate...",
  "wordCount": 730,
  "charCount": 4581,
  "tokensEstimate": 1146,
  "contentHash": "4f1c0e...",
  "chunks": [{ "index": 0, "text": "# 1. Whetting Your Appetite\n\n..." }],
  "crawledAt": "2026-10-22T02:30:11.000Z"
}

Failed and skipped URLs (with the reason) are listed in the RUN_SUMMARY record of the key-value store. They are not charged.

Pricing (pay per event)

You pay per page converted (a page that returned HTTP 2xx with at least minContentChars of text):

Apify plan	Price per 1,000 pages
Free	$2.00
Bronze (Starter)	$1.70
Silver (Scale)	$1.40
Gold (Business)	$1.00

Platform usage (compute, storage) is included; you don't pay it separately. Each run also has Apify's standard start fee of $0.00005 per GB of memory. Optional Apify datacenter proxy (or your own proxy URLs) is included in the price; residential proxy isn't supported. Set Max total charge on the run to cap your spend; the run stops cleanly when it's reached.

Limitations

• No JavaScript rendering. Sites that build their content in the browser (some SPAs, client-side blogs) may return little text; such pages are marked empty and not charged.
• Sites with strong bot protection (Cloudflare challenge, Akamai) may block requests; those pages are free. Enabling Apify datacenter proxy sometimes helps.
• PDFs and other files are skipped (free).

Responsible use

Only crawl public pages you are allowed to process. robots.txt is respected by default. Don't use the output to republish copyrighted content without permission.