Website to Markdown Crawler (llms.txt)
Pricing
from $1.50 / 1,000 crawled pages
Crawl a website or docs section into clean, LLM-ready markdown: sitemap first, then links, one row per page with title, depth, word and token counts, plus llms.txt and llms-full.txt files for AI agents and RAG. Polite, no browser needed. $1.50 per 1,000 pages.
Websites or site sections to crawl, one per line, e.g. https://docs.example.com or https://example.com/blog. Accounts on the Apify free plan can process up to 10 start URLs per run (20 pages each).
Pages with content to deliver per start URL (1–10,000). Pages without content, duplicates and filtered pages do not count.
How many links away from the start page to follow (0–20). Sitemap URLs count as depth 1; 0 reads only the start page.
auto: under the start URL's path when it has one, else the whole domain. path: the start URL's host, under its path. host: the start URL's host only. domain: every subdomain of the start URL's registrable domain.
Queue the site's sitemap URLs that are in scope before the links found on pages (sitemap from robots.txt or /sitemap.xml).
Deliver only pages whose URL matches one of these, one per line: a glob where * matches anything (e.g. /blog/) or a regular expression in slashes (e.g. //docs/v2//). Other pages are still read for their links. A regular expression that backtracks catastrophically is refused (prefer globs or character classes).
Do not deliver pages whose URL matches one of these (same syntax as the include patterns), e.g. /tag/ or /page/.
Markdown kept per page (1,000–200,000 characters), cut at a paragraph boundary. About 4 characters make one LLM token.
Write llms.txt (an index of the delivered pages) and llms-full.txt (all their markdown, up to 8 MB) per start URL to the run's key-value store. Free.
Safety cap on how many start URLs are crawled in this run; the rest are ignored and counted in the OUTPUT record.
