Website to Markdown Crawler for LLM & RAG
Pricing
from $1.00 / 1,000 page converteds
Crawl a website, sitemap or list of URLs and convert pages to clean Markdown for LLMs, RAG and AI agents. Readability main-content extraction, metadata, optional chunking. Fast HTTP crawler, $1 per 1,000 pages.
Pages or websites to convert. In crawl mode, links are followed from these pages.
Single pages: convert only the URLs you list. Crawl: follow links from the start URLs. Sitemap: take every URL from the site's sitemap.xml.
Maximum number of pages to convert in total. Each converted page is billed as one Page converted event.
How many link hops from the start URLs to follow (crawl mode).
Which links to follow: same hostname, same registered domain (incl. subdomains), or only URLs under the start URL's path (e.g. /docs/).
Glob patterns like https://example.com/docs/** or /regex/.
Main content uses Mozilla Readability (the Firefox Reader View engine) to keep the article/body and drop menus, footers, cookie bars and ads. Full page keeps everything except navigation, header, footer and sidebars.
Extra elements to remove before conversion, e.g. .promo, #related-posts.
Keep hyperlinks as Markdown links (absolute URLs). Off = plain text.
Split each page's Markdown into chunks of about this many characters for RAG / vector databases (heading- and paragraph-aware). 0 = no chunking. ~4 characters ≈ 1 token.
Characters repeated between consecutive chunks.
Pages with fewer words are skipped (not saved, not billed).
Add a text field without Markdown syntax.
Treat URLs that differ only in ?query as the same page.
Skip URLs disallowed by robots.txt for followed links.
