Website to Markdown – URL & Site Crawler for LLM/RAG
Pricing
from $0.40 / 1,000 page converted to markdowns
Convert any URL, sitemap or whole website to clean Markdown for LLMs, RAG and AI agents. Removes nav, footers and cookie banners, keeps headings, tables and code, adds RAG-ready chunks with token counts, llms.txt generation and an only-changed-pages mode. No browser: fast and cheap.
Pages to convert. In 'Same-domain crawl' mode these are the crawl entry points; in 'Sitemap' mode give a sitemap URL (e.g. https://example.com/sitemap.xml) or just the site URL and the sitemap is found via robots.txt.
[ { "url": "https://docs.apify.com/academy/scraping-basics-javascript" }, { "url": "https://www.paulgraham.com/greatwork.html" }]'Only these URLs' converts exactly the start URLs. 'Same-domain crawl' follows links on the same site up to Max depth. 'Sitemap' converts the URLs listed in the site's sitemap(s).
Hard cap on pages fetched in this run (start URLs included). Keeps cost predictable: at most maxPages x the page price.
Same-domain crawl only: how many clicks away from a start URL to follow (0 = start URLs only).
Only crawl / take from the sitemap URLs that match one of these globs, e.g. https://docs.example.com/guides/**. ** = anything, * = anything except '/'. Empty = all URLs on the domain.
[]Skip URLs matching any of these globs, e.g. /tag/, **/page/, **?replytocom=.
[]Split each page's Markdown into chunks of at most this many tokens (estimated as characters / 4) on heading and paragraph boundaries. Every chunk carries its heading path. 0 = no chunks. Included in the page price.
'Main content' removes navigation, headers, footers, sidebars, cookie banners and share widgets and keeps the article / docs body. 'Full page' converts the whole .
Render links as [text](absolute URL). Turn off for plain text-only Markdown (fewer tokens).
Render images as . Off by default to save tokens.
Add every absolute link found on the page (including PDFs and other files, which are listed but not fetched) to the dataset row.
Optional comma-separated CSS selectors removed before conversion, e.g. .newsletter-box, #comments.
Build an llms.txt index (llmstxt.org format: sections with - title: description) and an llms-full.txt with all Markdown, one pair per site, saved to the key-value store. Charged once per site.
For scheduled re-indexing: remembers each page's ETag / Last-Modified / content hash in a named key-value store and outputs (and charges) only pages that are new or changed since the previous run with the same start URLs.
Optional name for the change-tracking state (letters, digits, dashes). By default it is derived from the start URLs and crawl mode.
Skip URLs disallowed by robots.txt and honour Crawl-delay (max 10 s).
Parallel requests in total.
Parallel requests to one host. Keep it low to be polite.
Per-request timeout.
Retries for timeouts, connection errors, 429 and 5xx responses (with backoff).
Custom User-Agent header. Leave empty for the default, which identifies this Actor with a contact address.