Website to Markdown Crawler for LLM, AI & RAG
Pricing
from $1.50 / 1,000 converted pages
Crawl a website or URL list and return each page's main content as clean Markdown for LLMs and RAG. Use to build a knowledge base from docs or blogs. Input: startUrls, maxPages, optional followLinks, stayUnderStartPath, renderJs. Output per page: url, title, markdown, text, words. Obeys robots.txt.
Website start pages or a list of page URLs. Empty = a small example on crawlee.dev.
Stop after this many pages are saved (= pages you pay for). Default: 50.
Crawl links found on the pages. Off = convert only the start URLs (a URL list). Default: true.
Only follow links on the start URLs' websites (www. ignored). Default: true.
Only follow links under the start URL's folder, e.g. https://example.com/docs/ stays in /docs/. Default: false.
Link depth from the start pages (0 = start pages only). Default: 5.
Regular expressions; when set, only matching URLs are crawled. Example: /blog/
Regular expressions; matching URLs are skipped. Example: /tag/|/login
auto = render in a browser only when the HTML has almost no text (charged at the JS price); never = plain HTML only (cheapest); always = render every page. Default: "auto".
In auto mode, a page with fewer words than this whose HTML looks like a JavaScript app is rendered in a browser. Default: 50.
markdown = main content as markdown; text = plain text; html = raw HTML (large). Example: ["markdown", "text"]. Default: ["markdown", "text"].
[ "markdown", "text"]Politeness delay per host (0.5–30 s). A longer robots.txt Crawl-delay always wins. Default: 1.
Pages in progress at once across hosts (1–20). One host is always fetched one request at a time. Default: 4.