Website to Markdown Crawler for LLM, AI & RAG

Convert a website to markdown for LLM and RAG pipelines: a crawler that returns clean, structured text instead of raw HTML.

Point this Actor at a website or a list of pages. It returns each page's main content as clean markdown and plain text, ready for an LLM or a RAG index. It follows links on the same site up to the page limit you set, respects robots.txt, and never uses proxies to get around blocks. Plain HTML pages are cheap, and you pay the higher JavaScript price only when a page really needs a browser. Pages that fail or are blocked are listed and not charged.

Quick start for AI agents and API users

• What it does: Crawl a website or URL list and return each page's main content as clean Markdown for LLMs and RAG.
• When to use it: To build a knowledge base from docs or blogs.
• Example input (the form's prefill; it finishes in under a minute):

{
  "startUrls": [
    {
      "url": "https://crawlee.dev/"
    }
  ],
  "maxPages": 10,
  "sameDomainOnly": true,
  "renderJs": "never",
  "outputFormats": [
    "markdown",
    "text"
  ]
}

• Example output: one real dataset row from run tL1WyvhgDCRCO1BLh (2026-10-23), with long fields shortened and empty fields left out. Fields not shown are listed in the output table below.

{
  "url": "https://crawlee.dev/",
  "status": 200,
  "depth": 0,
  "renderedWithJs": false,
  "title": "Crawlee · Build reliable crawlers. Fast.",
  "description": "Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for ...",
  "language": "en",
  "canonicalUrl": "https://crawlee.dev/",
  "wordCount": 225,
  "markdown": "Crawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you.\n\n`npx crawlee ...",
  "text": "Crawlee is a web scraping library for JavaScript and Python. It handles blocking, crawling, proxies, and browsers for you.\nnpx crawlee cr...",
  "crawledAt": "2026-10-23T13:11:44+00:00"
}

• Cost (pay per event, Free-plan prices): Converted page $0.003; Converted JS-rendered page $0.006. Failed items are not charged.
• From an AI agent: with the Apify MCP server, call call-actor with actor phaux/website-to-markdown-crawler, or load it as its own tool with https://mcp.apify.com?tools=phaux/website-to-markdown-crawler. Read results with get-dataset-items.

What does Website to Markdown Crawler do?

• Crawls a site from one or more start URLs, following same-site links breadth-first up to Max pages and Max link depth, or converts just a URL list (followLinks: false).
• Extracts the main content (no menus, footers, cookie banners or ads) with trafilatura and returns markdown (headings, lists, tables, code blocks, links) and plain text, plus title, meta description, language, canonical URL and word count.
• Renders JavaScript only when needed: in auto mode a page is opened in headless Chromium only if its HTML has almost no text and looks like a JavaScript app.
• Polite by design: robots.txt (including Crawl-delay) is obeyed, one request at a time per host with a delay you set (default 1 s), honest bot User-Agent, no proxies, no CAPTCHA or anti-bot bypass. An HTTP 429 stops requests to that host for the rest of the run.
• Skips duplicates (/page vs /page/, identical content) and non-HTML files.

Why use it?

• Fixed price per page, not pay-per-compute: you know the cost before you run it.
• Cheap for normal websites: most pages are plain HTML and are charged at the HTTP price.
• Clean input for LLMs: markdown keeps the structure that chunkers and embedding pipelines need.
• Clear failure list: every page that failed, was blocked or was skipped is in FAILED_URLS with the reason, and none of them is charged.

Use cases: RAG knowledge bases from documentation sites, chatbot training data from your own help center, content audits and migrations, competitive content research on public pages, and feeding pages into GPT/Claude/Llama prompts.

How to use it

1. Add one or more Start URLs (e.g. https://docs.example.com/).
2. Set Max pages (this caps the cost) and, if needed, Stay under the start path or URL patterns.
3. Choose JavaScript rendering: auto (default), never (cheapest) or always.
4. Run, then export the dataset as JSON, JSONL, CSV or Excel, or read it via the API.

Input example

{
  "startUrls": [{"url": "https://crawlee.dev/"}],
  "maxPages": 10,
  "sameDomainOnly": true,
  "renderJs": "never",
  "outputFormats": ["markdown", "text"]
}

Field	Default	Description
startUrls	—	Start pages or a list of URLs. Empty = small example on crawlee.dev.
maxPages	50	Pages saved (and charged) at most.
followLinks	true	Crawl links; off = only the given URLs.
sameDomainOnly	true	Stay on the start URLs' websites.
stayUnderStartPath	false	Stay under the start URL's folder.
maxDepth	5	Link depth from the start pages.
includeUrlPatterns / excludeUrlPatterns	—	Regular expressions to scope the crawl.
renderJs	auto	auto, never or always.
minWordsBeforeJs	50	Auto mode threshold.
outputFormats	markdown, text	Any of markdown, text, html.
includeLinksInMarkdown	true	Keep [text](url) links.
requestIntervalSecs	1	Delay between requests to one host (0.5–30 s).
maxConcurrency	4	Pages in progress at once (one per host at a time).
timeBudgetSecs	0	Stop after N seconds and keep what is saved.

Output example

{
  "url": "https://crawlee.dev/python/docs/quick-start",
  "finalUrl": null,
  "status": 200,
  "depth": 1,
  "renderedWithJs": false,
  "title": "Quick start | Crawlee for Python · Fast, reliable Python web crawlers.",
  "description": "...",
  "language": "en",
  "canonicalUrl": "https://crawlee.dev/python/docs/quick-start",
  "wordCount": 954,
  "markdown": "# Quick start\n\nThis short tutorial will help you start scraping with Crawlee in just a minute or two. ...",
  "text": "Quick start\nThis short tutorial will help you start scraping with Crawlee ...",
  "crawledAt": "2026-10-22T01:20:00+00:00"
}

FAILED_URLS (key-value store) lists pages not saved, with a code: robots_disallowed, blocked (401/403), rate_limited (429), http_error, not_html, no_content, duplicate, out_of_scope or max_charge_reached. RUN_STATS has the counts.

Pricing

Pay per saved page:

Event	Free	Bronze	Silver	Gold+
Converted page (plain HTML)	$0.003	$0.002	$0.0017	$0.0015
Converted JS-rendered page	$0.006	$0.005	$0.004	$0.003
Actor start	$0.00005	$0.00005	$0.00005	$0.00005

Example: 1,000 documentation pages (plain HTML) on the Bronze plan cost about $2.00. Set a maximum total charge on the run and the crawl stops before it.

Tips and limits

• Use renderJs: never for static sites and docs; it is the cheapest and fastest.
• Large sites: combine Max pages with Stay under the start path or include patterns.
• Login-only pages, paywalls and pages behind anti-bot walls are not crawled; they show up as blocked (free).
• The Actor reads only public pages. You are responsible for following each website's terms and for any personal data on the pages you collect.

FAQ

Can I convert only a list of pages without crawling? Yes. Put the pages in Start URLs and set followLinks to false.

Does it render JavaScript? In auto mode (the default) a page is opened in headless Chromium only when its HTML has almost no text and looks like a JavaScript app. Use never for the cheapest runs on static sites, or always to render every page.

How do I limit the cost? Set Max pages (pages saved and charged at most) and, if you like, a maximum total charge for the run. The crawl stops before it.

Are failed or blocked pages charged? No. They are listed in FAILED_URLS with a reason and are free.

Does it use proxies or get around anti-bot walls? No. It obeys robots.txt, sends an honest User-Agent and does not bypass logins, paywalls or CAPTCHAs.

Which output formats can I export? Each page has markdown and text (and html if you ask for it). Export the dataset as JSON, JSONL, CSV or Excel, or read it through the API.

Something isn't working? Open an issue on the Actor's Issues tab with the run ID.