Website to Markdown Crawler for LLM & RAG (llms.txt)

Crawl a website, a docs section or a blog and get every page as clean Markdown, ready for LLMs, RAG pipelines, vector databases and AI agents. Menus, cookie banners, sidebars and footers are removed; headings, lists, tables and code blocks are kept. Optional RAG chunks, llms.txt and llms-full.txt come in the same run.

• Flat, predictable price: $1 per 1,000 pages. No browser and no compute bill that depends on how heavy a site is. You know the cost before you start: maxPages × $0.001.
• Main-content extraction with trafilatura, plus a fallback to the page's own <article>/<main> when a page is a structured reference (API docs, changelogs) that generic extractors flatten.
• Stays where you point it. https://example.com/docs/ crawls the docs and nothing else. Glob patterns widen or narrow the scope.
• Finds pages menus don't link to by reading the site's sitemap.xml (optional).
• No duplicates. Redirects, rel=canonical, #fragments and utm_* tracking parameters are collapsed into one page.
• Polite. Respects robots.txt by default, retries 429/5xx with back-off, sends an honest User-Agent.
• llms.txt generator. Get an llmstxt.org index of the site and an llms-full.txt with every page's Markdown, free.
• Failed and empty pages cost nothing.

Who it's for

• AI engineers loading documentation, help centres or knowledge bases into a vector database for a chatbot or RAG search.
• Agent and MCP builders who need "read this site" as one cheap tool call.
• SEO and content teams who want a site's text in one file, or an llms.txt for their own site.
• Researchers archiving blogs and docs as plain text.

Quick start

1. Paste one or more start URLs, for example https://docs.example.com/.
2. Set Max pages (you pay only for pages saved, so this also caps the cost).
3. Run. Each page becomes a dataset row with markdown; llms.txt and llms-full.txt land in the run's key-value store.

Input

Field	Default	Description
startUrls		Where the crawl starts. Links are followed only under each start URL's path.
maxPages	100	Stop after this many saved pages.
maxCrawlDepth	20	Links away from a start URL. 0 = only the start URLs. Sitemap pages count as depth 1.
useSitemaps	false	Also queue every in-scope URL from the site's sitemaps (found via robots.txt, else /sitemap.xml).
includeUrlGlobs		Crawl only URLs matching one of these globs, instead of "under the start URL". ** matches anything, * anything but /, {a,b} alternatives.
excludeUrlGlobs		Never crawl URLs matching these globs, e.g. **/tag/**.
respectRobotsTxt	true	Skip pages robots.txt disallows. Turn off only for your own site.
maxConcurrency	10	Pages fetched at once.
contentMode	main	main: article/docs text only. full: everything in <body>, navigation and footer included.
includeLinks	false	Keep links as [text](url).
includeImages	false	Keep images as .
saveText	false	Add a plain-text text field.
saveHtml	false	Add the original html.
generateLlmsTxt	true	Write llms.txt and llms-full.txt to the key-value store.
detectLanguage	true	Detect language from the text when the page doesn't declare it.
chunkSize	0 (off)	Split each page's Markdown into chunks of about this many characters for embeddings. 1000–2000 suits most models.
chunkOverlap	200	Characters shared by neighbouring chunks.

{
  "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
  "maxPages": 300,
  "useSitemaps": false,
  "excludeUrlGlobs": [{"glob": "**/genindex*"}],
  "chunkSize": 1500,
  "chunkOverlap": 200
}

Output

One row per saved page. A field without a value is null.

Field	Description
url	The page's URL after redirects.
requested_url	The URL that was requested, if it redirected.
title, description	From <title> and the meta description (Open Graph as fallback).
language	From <html lang>, else detected from the text (ISO 639-1, e.g. en, de).
author, published_at	From meta tags or <time datetime>, when the page has them.
canonical_url	The page's rel=canonical.
depth, referrer_url	Links away from the start URL, and the page the link was found on (sitemap for sitemap pages).
http_status	Usually 200.
markdown	The page content as Markdown.
text, html	Only when saveText / saveHtml is on.
word_count, char_count	Size of markdown.
chunks	When chunkSize > 0: [{"index", "start", "end", "text"}], offsets into markdown.
truncated	true if a giant page was cut at 4 million characters.
crawled_at	UTC timestamp.
error	Only on rows for start URLs that failed (HTTP 404, blocked by the site's robots.txt, no text content found (the page may need JavaScript to render)). Dead links found during the crawl are counted in the log instead of filling your dataset.

Example row from a real run on docs.python.org (markdown and chunks shortened):

{
  "url": "https://docs.python.org/3/tutorial/classes.html",
  "requested_url": null,
  "title": "9. Classes — Python 3.15.1 documentation",
  "description": "Classes provide a means of bundling data and functionality together. Creating a new class creates a new type of object, allowing new instances of that type to be made. Each class instance can have ...",
  "language": "en",
  "author": null,
  "published_at": null,
  "canonical_url": "https://docs.python.org/3/tutorial/classes.html",
  "depth": 2,
  "referrer_url": "https://docs.python.org/3/tutorial/index.html",
  "http_status": 200,
  "markdown": "# 9. Classes

Classes provide a means of bundling data and functionality together.  Creating
a new class creates a new *type* of object, ...",
  "text": null,
  "html": null,
  "word_count": 5420,
  "char_count": 34733,
  "chunks": [{"index": 0, "start": 0, "end": 1037, "text": "# 9. Classes

Classes provide a means of bundling data ..."}],
  "truncated": false,
  "crawled_at": "2026-10-24T02:58:55+00:00",
  "error": null
}

The dataset has four views: Pages, RAG chunks, Metadata and All fields. Export as JSON, JSONL, CSV or Excel, or read rows from the API:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("malmurk/website-to-markdown-crawler").call(run_input={
    "startUrls": [{"url": "https://docs.example.com/"}], "maxPages": 500, "chunkSize": 1500,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page["chunks"] or []:
        print(page["url"], chunk["text"][:80])  # -> your embedding model + vector DB

llms_full = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("llms-full.txt")["value"]

llms.txt and llms-full.txt

With generateLlmsTxt on, the run's key-value store gets two files (links in the run's Output tab):

• llms.txt: # Site title, > description, then - [Page title](url): description for every saved page, sorted by URL.
• llms-full.txt: every page's Markdown in one file, each under # Title and Source: url. Paste it into a chat, or drop it into a long-context model.

Pricing

Pay per event:

Event	Price	Charged for
page	$1 per 1,000 pages ($0.001 each)	Every page saved to the dataset with content.
Actor start	$0.00005 per run	Once per run (per GB of memory).

• Pages that fail (404, timeouts, DNS errors), are blocked by robots.txt, are files (PDF, images, JSON) or have no text cost nothing.
• Duplicates (redirects, canonical copies, tracking-parameter variants) are skipped before they are saved, so you never pay twice for one page.
• llms.txt, llms-full.txt, chunks, language detection and metadata are included.
• The run stops cleanly at the maximum charge you set for it.

A 300-page docs site costs $0.30. A 5,000-page site costs $5.

FAQ

Does it render JavaScript? No. It reads the HTML the server sends, which is what makes it fast and cheap. Server-rendered sites, static site generators (Docusaurus, MkDocs, Sphinx, Hugo, Jekyll, Next.js/Nuxt with SSR, WordPress) and most docs, blogs and help centres work. A single-page app that renders everything in the browser returns an empty page; you then get an error row for that start URL and pay nothing.

Why did it stop before maxPages? Every in-scope page was crawled. Check the scope: https://example.com/docs/intro only crawls pages under /docs/intro. Start from https://example.com/docs/ (or the site root) to crawl more, turn on useSitemaps, or add includeUrlGlobs.

Can it crawl sites that block bots? It does not use proxies or try to bypass blocking, CAPTCHAs or logins. A site that returns 403 to it is respected. Crawl only content you are allowed to use.

How is this different from a browser-based crawler? Browser crawlers bill compute per second of Chrome time, so the same 1,000 pages can cost very different amounts on different sites. This Actor costs $1 per 1,000 saved pages on every site. Measured on Apify: about 1–1.5 pages per second at the default 1 GB memory; more memory crawls faster at the same price.

Can I keep links and images? Yes: includeLinks and includeImages. They are off by default because plain text embeds better.

What does main-content mode remove? Navigation, headers, footers, sidebars, cookie notices, share buttons and comments. If you need everything on the page, use contentMode: full.

Is there a size limit per page? Pages over 3 MB of HTML (rare: giant changelogs and indexes) are skipped and free. Markdown over 4 million characters is cut and flagged truncated.

Limitations

• No JavaScript rendering (see FAQ). No login, cookies or proxies.
• Only HTML pages are converted. Links to PDFs, images, archives and other files are not followed. For PDFs, use PDF & Document Text Extractor.
• Main-content detection is heuristic. On unusual layouts compare with contentMode: full.
• Code blocks are kept, but blank lines inside them can be dropped.

How it works

HTTP with httpx, main content with trafilatura (Apache-2.0), page conversion with markdownify (MIT), robots.txt with Protego (BSD-3), language detection with py3langid (BSD-3).

Local development

python -m venv .venv && .venv/Scripts/pip install -r requirements-dev.txt   # Linux/macOS: .venv/bin/pip
python -m pytest -q
docker build -t website-to-markdown-crawler . && docker run --rm -v "$PWD/INPUT.json:/usr/src/app/INPUT.json" website-to-markdown-crawler