Website to Markdown for LLM & RAG — Pages or Whole Sites

Convert any web page — or a whole website, blog or docs portal — into clean Markdown ready for ChatGPT, Claude, LangChain, LlamaIndex and vector databases.

Only the main content is kept (Mozilla Readability): no menus, cookie banners, footers or ads. Tables become GFM tables, links and images become absolute URLs, and each page comes with title, description, author, publish/modify dates, language, canonical URL and word count.

How to run it in 3 steps

1. Add your input. In URLs paste page or site URLs (turn on Crawl for whole sites) — e.g. https://blog.apify.com/what-is-web-scraping/. The form is already filled with a working example, so you can skip this step for a first try.
2. Press Start. The prefilled run returns 1 page in about 2 s and costs about $0.003. Set Max results (cost cap) to cap the cost of bigger runs; you only pay for pages actually delivered.
3. Get the data. Open the Output tab and export JSON, CSV or Excel — or run it from the API, Make, Zapier, n8n or an AI agent (see below).

Features

• HTTP only, no browser: plain HTTP with an in-memory frontier — no browser, no request-queue writes. 40 docs pages in ~6 s.
• Crawl or list: convert a list of URLs, or crawl a site breadth-first with depth / per-site caps, regex include/exclude, and optional sitemap.xml discovery.
• Two extraction modes: Article (Readability, best for blogs/news/docs) or Full page (whole body minus nav/header/footer/sidebars).
• RAG-friendly output: one row per page with markdown + metadata, so you can chunk and embed straight away.
• Pay per page: $3 per 1,000 converted pages. Failed pages, non-HTML files and pages below minWordCount are free.

Use cases

• Build a knowledge base / RAG index from documentation or a help center.
• Feed competitor blogs or news to an LLM for summarization.
• Create fine-tuning or evaluation datasets from web content.
• Archive articles as Markdown for Obsidian / Notion.

Input

{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "crawl": true,
  "maxDepth": 2,
  "maxPagesPerSite": 100,
  "includeUrlPattern": "/docs/",
  "extractionMode": "article",
  "minWordCount": 50
}

Output

Real output (default input)

First 2 of 1 records from a run with the prefilled input (2.5 s on a test machine; long values shortened here):

[
  {
    "url": "https://blog.apify.com/what-is-web-scraping/",
    "title": "What is web scraping?",
    "description": "The basics of web scraping: what it is, how it works, real-world use cases, and how to get started.",
    "author": "David Barton",
    "publishedAt": "2024-10-15T14:32:00.000Z",
    "modifiedAt": "2026-10-09T17:13:43.000Z",
    "siteName": "Apify Blog",
    "language": "en",
    "canonicalUrl": "https://blog.apify.com/what-is-web-scraping/",
    "image": "https://storage.ghost.io/c/f2/6e/f26ec999-9a90-4aee-a0d4-9b3ca2bb668f/content/images/2024/04/what-is-web-scraping-complete-guide.png",
    "markdown": "Web scraping is the process of automatically extracting data from a website. You use a program called a web scraper to access a web page, in…",
    "wordCount": 1428,
    "extraction": "article",
    "depth": 0,
    "httpStatus": 200,
    "scrapedAt": "2026-10-15T12:09:14.663Z"
  }
]

{
  "url": "https://crawlee.dev/js/docs/introduction/setting-up",
  "title": "Setting up | Crawlee for JavaScript",
  "description": "…",
  "author": null,
  "publishedAt": null,
  "language": "en",
  "markdown": "To run Crawlee on your own computer, you need to meet the following pre-requisites first:\n\n1. Have **Node.js version 16.0** …",
  "wordCount": 407,
  "extraction": "article",
  "depth": 1,
  "httpStatus": 200,
  "scrapedAt": "2026-10-14T12:00:00.000Z"
}

Pricing

• $3.00 per 1,000 pages ($0.003 per page).
• The prefilled input returns 1 pages → $0.003.
• Example: 1,000 pages → $3.00; 10,000 → $30.00. Set Max results (cost cap) to limit any run.

• Failed pages, non-HTML files and pages below minWordCount are free.
• Your Apify spending limit is respected exactly: the actor stops charging and finishes cleanly when it is reached. Apify platform usage (compute, proxy) is included in the price.

Use with AI agents (MCP), LangChain, RAG & the API

This actor is a ready-made tool for AI agents: an LLM can call it, wait for the run and read clean JSON back.

MCP server (Claude Desktop, Claude Code, Cursor, VS Code, any MCP client) — add Apify's hosted MCP server with this actor enabled as a tool:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=torix33/website-to-markdown",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

LangChain / LangGraph (pip install langchain-apify, set APIFY_API_TOKEN):

from langchain_apify import ApifyActorsTool, ApifyWrapper
from langchain_core.documents import Document

# 1) give the actor to an agent as a tool
tool = ApifyActorsTool("torix33/website-to-markdown")

# 2) or load results as Documents for a RAG index
loader = ApifyWrapper().call_actor(
    actor_id="torix33/website-to-markdown",
    run_input={"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "crawl": True, "maxDepth": 2, "maxPagesPerSite": 100, "includeUrlPattern": "/docs/", "extractionMode": "article", "minWordCount": 50},
    dataset_mapping_function=lambda item: Document(page_content=item["markdown"], metadata={"url": item.get("url")}),
)
docs = loader.load()

REST API — one synchronous call, dataset items in the response:

curl -X POST "https://api.apify.com/v2/acts/torix33~website-to-markdown/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "crawl": true, "maxDepth": 2, "maxPagesPerSite": 100, "includeUrlPattern": "/docs/", "extractionMode": "article", "minWordCount": 50}'

Also works with the Apify Python/JS clients, Make, Zapier, n8n, Google Sheets, webhooks and schedules. Export as JSON, CSV, Excel, XML or RSS.

RAG tip: Each row is one page with markdown, title and url — split markdown by headings, embed, and store url for citations. Need ready-made chunks? Use Website to RAG Chunks.

Tips

• Use maxResults as a hard cost cap.
• Sites that block cloud servers: enable Apify Proxy in Proxy configuration.
• JavaScript-only single-page apps (content rendered in the browser) may return little text — the actor is HTTP-only for speed and price.

Limitations

• HTTP only: single-page apps that render content in the browser may return little or no text.
• Login-protected pages are not supported; PDFs and other non-HTML files are skipped (free).
• A crawl stops at maxPagesPerSite / maxDepth; very large sites need several runs or a URL pattern.

FAQ

Is it a Firecrawl or Jina Reader alternative? Yes, for static and server-rendered sites: it crawls and returns clean Markdown per page at $3 per 1,000 pages, without a browser.

Can it crawl a whole documentation site? Yes. Set crawl: true, maxDepth, maxPagesPerSite, includeUrlPattern, and optionally use sitemap.xml.

Does it work on JavaScript-only sites? Single-page apps that render everything in the browser may return little text; the actor is HTTP-only for speed and price.