Website to Markdown for LLMs & RAG — Clean Content Extractor

Turn any URL — or a whole site — into clean, LLM-ready Markdown, plain text, or structured JSON. Navigation, ads, cookie bars and boilerplate are stripped; headings, links, tables and code blocks survive intact. Drop the output straight into a prompt, a vector store, or a fine-tuning set.

You pay only per page actually extracted. No startup fee — most alternatives charge one on every run. Failed pages cost $0.

Quick start

{
  "urls": ["https://en.wikipedia.org/wiki/Large_language_model"]
}

→ one dataset item with clean Markdown plus every metadata field you'd want:

{
  "url": "https://en.wikipedia.org/wiki/Large_language_model",
  "title": "Large language model",
  "description": "A large language model (LLM) is a neural network trained on…",
  "author": "", "publishedDate": "", "siteName": "Wikipedia", "lang": "en",
  "wordCount": 13299,
  "markdown": "A **large language model** (**LLM**) is a [neural network](…) …",
  "links": [{ "href": "…", "text": "…" }]
}

Crawl an entire site — the fast, polite way

{
  "urls": ["https://docs.example.com"],
  "crawl": true,
  "useSitemap": true,
  "maxPages": 200,
  "maxConcurrency": 8,
  "includeUrlPatterns": ["*/docs/*"],
  "excludeUrlPatterns": ["*/changelog/*", "*.pdf"]
}

• Sitemap discovery seeds the crawl from the site's own sitemap.xml (including Sitemap: entries in robots.txt) — the fastest, most complete way to reach every page, no link-hopping required.
• Include/exclude glob filters keep the crawl (and your bill) exactly on the content you want.
• Concurrent fetching (up to 15 parallel) with per-run page caps and depth limits, so cost is always bounded and predictable.
• robots.txt respected by default.

Features

Output formats	Markdown (GFM: tables, fenced code), plain text, cleaned HTML, or JSON with everything
Main-content extraction	Readability isolates the article; or convert the full page body — your choice
JS rendering	Real Chromium for SPAs/React/Vue; switch it off for a much faster static fetch
Metadata	title, description, author, published date, site name, language, OG image, word count
Crawling	Same-domain BFS + sitemap seeding, URL glob filters, max pages/depth/concurrency
Clean-up	Remove any elements by CSS selector before extraction
Reliability	Optional Apify Proxy (incl. residential) for bot-protected sites
Politeness	robots.txt respected by default; bounded concurrency

Built for AI pipelines

• RAG ingestion: crawl your docs/blog → chunk the markdown field → embed. Re-run on a Schedule to keep the index fresh.
• Agent tool: call via the Apify MCP server or API and give your agent clean page content instead of raw HTML soup.
• Dataset building: sitemap + filters → every article on a site as uniform Markdown records with metadata.
• Content migration: legacy CMS → Markdown files in one run.

curl -X POST "https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://blog.example.com"], "crawl": true, "useSitemap": true, "maxPages": 50}'

Pricing

A small fixed price per successfully extracted page, plus standard Apify platform usage for the compute your run consumes. No startup fee (most alternatives charge $0.0015–$0.09 per run before extracting anything). Failed pages are never billed. Tip: turn off jsRendering on static sites — runs get several times faster and cheaper.

FAQ

How is this different from a raw scraper? You get reading-quality content: boilerplate stripped, structure preserved, ready for a model — not a DOM dump.

Does it respect site owners? Yes — robots.txt is honored by default, sitemap seeding uses the URL list the site itself publishes for crawlers, and concurrency is bounded. Extraction is user-directed: you choose the URLs, and you're responsible for complying with the target site's terms and applicable law.

JavaScript-heavy site returns thin content? Keep jsRendering: true (default) and try waitUntil: "networkidle".

Blocked by anti-bot? Enable Apify residential proxy in the input.

---

Missing a feature you need? Open an issue — this actor ships improvements fast. If it saves you time, a ⭐ review helps others find it.

More tools by Runlayer

Part of a suite of fast, no-nonsense web utilities — all pay-per-result, charged only on success, no startup fee:

• Website Screenshot API — full-page PNG / JPEG / WebP / PDF
• Website to Markdown — clean Markdown for LLMs & RAG
• PDF & DOCX to Markdown — documents → Markdown for RAG
• SEO Audit + Core Web Vitals — scored on-page audit with fixes
• Website Performance Audit — bulk Core Web Vitals & page speed
• Tech Stack Detector — CMS, frameworks, analytics, hosting
• Domain, DNS & WHOIS Lookup — records, registration, SSL
• RSS Feed Reader — RSS / Atom / JSON → normalized JSON
• Job Postings API — Greenhouse, Lever, Ashby & more

Use with AI agents (MCP)

This Actor is callable by AI agents through the Apify MCP server. Agents in Claude, Cursor, Windsurf, LangGraph, CrewAI and others can discover it via search-actors and run it as a tool — its inputs and outputs are fully described in the schema for reliable agent use.