Website to Markdown Crawler for LLM & RAG

Feed any website or docs site to ChatGPT, Claude or your RAG pipeline as clean Markdown, for $1 per 1,000 pages (plus a $0.005 start fee).

Who it's for

AI engineers building RAG and chatbots, developers giving agents web access, and teams turning docs, help centres or pricing pages into a knowledge base.

What people use it for

• 📚 Docs to Markdown for ChatGPT or Claude. Crawl a documentation section (e.g. /tutorial/) and upload it as one knowledge file.
• 🧠 RAG ingestion. One item per page with title, canonical URL, language and word count, ready to chunk and embed.
• 💬 Help-centre chatbots. Turn support articles into a clean corpus for a customer-support bot.
• 💷 Competitor pricing pages. Snapshot pricing pages as Markdown that an LLM can compare.

Real sample output

From a real run on https://docs.python.org/3/tutorial/index.html (shortened):

{
  "url": "https://docs.python.org/3/tutorial/index.html",
  "title": "The Python Tutorial",
  "siteName": "Python documentation",
  "language": "en",
  "renderedWith": "http",
  "wordCount": 890,
  "markdown": "# The Python Tutorial\n\nTip\n\nThis tutorial is designed for *programmers* that are new to the Python language, **not** *beginners* who are new to programming.\n\nPython is an easy to learn, powerful programming language. It has efficient high-level data structures…"
}

What it costs (real prices)

Prices are pay per event: $0.001 per page, +$0.0008 only if a page needs headless Chrome, and a $0.005 start fee per run (1 GB default memory). Failed pages are free.

Job	Calculation	USD	GBP*
100 plain HTML pages	100 × $0.001 + $0.005	$0.105	£0.08
1,000 plain HTML pages	1,000 × $0.001 + $0.005	$1.005	£0.75
500-page docs site, 100 need Chrome	500 × $0.001 + 100 × $0.0008 + $0.005	$0.585	£0.44

*GBP at 1 USD = £0.74711 (ECB rate, 6 Nov 2026). Apify bills in USD.

Integrations

All recipes below use Apify's own official integrations. You need a free Apify account.

• Zapier. Add Apify as the action in a Zap, pick Run Actor (or Run Task for a saved example), choose ryumoon-data/website-to-markdown-rag and paste the input JSON. Then add Fetch dataset items to pass the results to Google Sheets, Slack, Notion and so on. Synchronous runs in Zapier time out after 30 seconds, so for big batches use the Finished Actor run trigger in a second Zap. Apify docs: Zapier
• Make. Add the Apify module Run an Actor with "Run synchronously" set to Yes, then Get Dataset Items using the run's default dataset ID, then e.g. Google Sheets "Bulk Add Rows". For long runs, use the Watch Actor Runs trigger instead. Apify docs: Make
• n8n. Install the Apify node (@apify/n8n-nodes-apify from Community Nodes on self-hosted, or search "Apify" in the nodes panel on n8n Cloud). Add an Apify API key or OAuth2 (Cloud) credential, then use it as an action to run this Actor, or as the On new Apify Event trigger to process results when a run finishes. Apify docs: n8n
• MCP (Claude, ChatGPT and other AI agents). Add Apify's hosted MCP server to your client. To load only this Actor, use https://mcp.apify.com?tools=ryumoon-data/website-to-markdown-rag. The first time you connect, your browser opens so you can sign in to Apify. Alternatively, send an Authorization: Bearer <APIFY_TOKEN> header:

  { "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=ryumoon-data/website-to-markdown-rag" } } }
  

  Your assistant can then run it from a prompt like "Convert https://fastapi.tiangolo.com/tutorial/ and its subpages to Markdown". Apify docs: MCP server

---

Turn any web page, or a whole docs site, into clean Markdown that LLMs understand. Give it URLs and get back one item per page: tidy Markdown (headings, lists, tables, code blocks, links), the title, description, author, publish date, language and canonical URL. Menus, cookie banners, footers and ads are stripped out, so you don't waste tokens on boilerplate.

Every page is fetched with plain HTTP first, and headless Chrome starts only when a page really needs JavaScript or shows a bot check. You pay $1 per 1,000 pages, plus a small extra only for the pages that needed a browser.

Use with AI agents (MCP, Claude, ChatGPT, Cursor)

Purpose: Turn any web page, or a whole website, into clean Markdown text for LLMs, RAG pipelines and AI agents, with title, metadata and links. Renders JavaScript only when needed.

When to use it

• The agent needs the readable content of one or more URLs (docs, articles, product pages) as Markdown.
• Building a knowledge base or RAG index from a whole site or a docs section (set maxCrawlDepth and includeUrlPatterns).

When not to use it

• For structured fields such as prices or contact details, use a dedicated scraper (e.g. Company Contact Finder).
• Pages behind a login are not supported.

Example input

{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy/web-scraping-for-beginners"
    }
  ],
  "maxCrawlDepth": 1,
  "maxPages": 20,
  "mainContentOnly": true
}

Output fields (one dataset item per row): url, title, markdown, wordCount, description, language, publishedAt, statusCode, renderedWith (http/browser), error.

Cost: $0.005 per run + $0.001 per page ($1 per 1,000 pages). Cap spend with maxPages.

Connect it to your agent: add this remote MCP server (Apify's official MCP server, loaded with just this tool):

{"mcpServers": {"website-to-markdown-rag": {"url": "https://mcp.apify.com?tools=ryumoon-data/website-to-markdown-rag"}}}

Or call it from any agent with the Apify API: POST https://api.apify.com/v2/acts/ryumoon-data~website-to-markdown-rag/run-sync-get-dataset-items with the input above as the JSON body. Agents without an Apify account can pay per run with agentic payments (x402, MPP or Skyfire).

What can it do?

• 🧹 Clean Markdown: main content only (Mozilla Readability), with GitHub-style tables and fenced code blocks.
• ⚡ HTTP first, browser only when needed: in auto mode Chrome is used only for JavaScript apps (React/Vue/Next shells) and pages with bot checks.
• 🕸️ Crawl a site: follow links on the same website to a depth and page limit you choose, with include/exclude regex filters (great for /docs/ or /blog/).
• 🧾 Metadata for citations: title, description, author, published/modified date, language, canonical URL, site name and preview image.
• ✂️ LLM-friendly options: drop links or images, cap Markdown length per page, remove your own CSS selectors, or get plain text or cleaned HTML instead.
• 🤖 Works with AI agents: like every Apify Actor, call it from the Apify API or Apify's official integrations: MCP server, LangChain, LlamaIndex, n8n, Make and Zapier.

Use cases

• RAG pipelines: load documentation, help centres or blogs into a vector database (Pinecone, Qdrant, Weaviate and others).
• AI agents and chatbots: give your agent a "read this URL" tool.
• Fine-tuning and evals: collect clean text datasets from public pages.
• Content monitoring and summaries: fetch articles daily and summarise them with an LLM.
• SEO and content audits: word counts, titles, descriptions and canonical URLs in one table.

How much does it cost?

Pay-per-event pricing. Failed pages are free.

Event	Price	When it is charged
Actor start	$0.005	Once per run (at the default 1 GB memory)
Web page	$0.001	Each page converted to Markdown ($1 per 1,000)
Browser render	$0.0008	Extra, only for pages that needed headless Chrome

Examples:

• 1,000 normal HTML pages (docs, blogs, news): about $1.01
• 100 pages from a JavaScript-heavy app: about $0.19
• One quick page for an agent: $0.006

Set renderingMode to http to never pay the browser extra, and use Max pages or Apify's maximum cost per run to cap spend.

Input

Field	What it does	Default
startUrls (required)	Pages to convert	two example pages
renderingMode	auto, http or browser	auto
mainContentOnly	Strip navigation, footers, sidebars	true
outputFormats	Any of markdown, text, html	["markdown"]
includeLinks / includeImages	Keep links / images in Markdown	true / false
maxMarkdownChars	Truncate long pages (0 = no limit)	0
removeSelectors	Extra CSS selectors to delete	none
maxCrawlDepth	0 = only your URLs; 1+ = follow same-site links	0
maxPagesPerStartUrl / maxPages	Crawl limits	10 / 100
includeUrlPatterns / excludeUrlPatterns	Regex filters for followed links	none
includePageLinks	Add a links array to each item	false
requestTimeoutSecs, maxConcurrency, proxyConfiguration	Advanced	30, 8, none

Crawl the Apify docs' academy section, 50 pages, no links in the Markdown:

{
  "startUrls": [{ "url": "https://docs.apify.com/academy" }],
  "maxCrawlDepth": 2,
  "maxPagesPerStartUrl": 50,
  "includeUrlPatterns": ["/academy/"],
  "includeLinks": false
}

Output

One dataset item per page (Markdown shortened):

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "finalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "statusCode": 200,
  "title": "Retrieval-augmented generation - Wikipedia",
  "description": null,
  "language": "en",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "renderedWith": "http",
  "depth": 0,
  "markdown": "# Retrieval-augmented generation - Wikipedia\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information...",
  "wordCount": 2218,
  "markdownTruncated": false,
  "crawledAt": "2026-11-02T17:40:00.000Z",
  "error": null
}

Pages that fail (404, timeouts, blocked) are still listed with error set, and you are not charged for them. A SUMMARY record in the key-value store has the run's counts.

Export the dataset as JSON, JSONL, CSV or Excel, or read it straight from the API: GET https://api.apify.com/v2/datasets/{datasetId}/items?format=json.

Using it from an AI agent

• Apify MCP server: add this Actor to your MCP client (Claude, Cursor, VS Code and others) and the agent can call it with {"startUrls":[{"url":"..."}]}.
• API, synchronous: POST https://api.apify.com/v2/acts/ryumoon-data~website-to-markdown-rag/run-sync-get-dataset-items?token=YOUR_TOKEN with the input JSON as the body returns the items directly.
• LangChain / LlamaIndex: use the ApifyWrapper / Apify dataset loader, mapping markdown to the document text and url to the source.

Run it from code (Python) and load into LangChain

pip install apify-client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("ryumoon-data/website-to-markdown-rag").call(run_input={
    "startUrls": [{"url": "https://docs.python.org/3/tutorial/index.html"}],
    "maxCrawlDepth": 1,
    "maxPages": 30,
    "includeUrlPatterns": ["/3/tutorial/"],
    "includeLinks": False,
})
pages = [p for p in client.dataset(run.default_dataset_id).iterate_items() if not p.get("error")]
print(len(pages), "pages")

LangChain (pip install langchain-apify): load a finished run's dataset with ApifyDatasetLoader(dataset_id=..., dataset_mapping_function=lambda p: Document(page_content=p["markdown"], metadata={"source": p["url"]})). See Apify docs: LangChain.

Chunking tips for RAG

• Split on ## headings first, then by size; keep code fences whole.
• Store url, title and the heading path as metadata so answers can cite the page.
• Set includeLinks: false for shorter embeddings; set maxMarkdownChars to cap huge pages.
• Re-run on a schedule (Apify Schedules) to keep the index fresh.

Tips and limits

• Sites with heavy bot protection (DataDome, PerimeterX, some Cloudflare setups) may still block the request. Those pages are returned with an error and not charged. You can add your own proxy URLs.
• Apify residential proxy is not supported, to keep the per-page price low.
• PDFs, images and other binary files are skipped.
• Please respect each website's terms and robots rules, and only collect content you're allowed to use.

FAQ

How is this different from Apify's RAG Web Browser? RAG Web Browser starts from a Google search and fetches the top results. This Actor converts URLs you already have (or crawls a site from them), doesn't search, and only starts a browser for pages that need one.

Can I get plain text instead of Markdown? Yes: set outputFormats to ["text"] (or several formats at once).

Is it legal? It reads public web pages like a browser does. You're responsible for how you use the content, including copyright and personal data rules.