Website to Markdown — Content Crawler for LLM & RAG
Pricing
from $0.60 / 1,000 results
Content Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.
Enter the page(s) to start crawling from, e.g. https://docs.apify.com/platform. The crawler follows links from here in breadth-first order.
[ { "url": "https://docs.apify.com/platform" }]Enter the maximum number of pages to crawl and convert, e.g. 50. This is the billed unit ($1 per 1,000 pages) — the crawl stops exactly at this count.
Enter how many links deep to follow from a start URL, e.g. 3. Set 0 to crawl only the start URL(s).
Turn this on to only follow links on the same domain as the start URL (www. is treated as the same domain), and off to also follow links to other domains.
Optional. Only crawl URLs whose path starts with one of these prefixes, e.g. /docs. Leave empty to crawl every path allowed by the other settings.
[]Optional. Regular expressions tested against the full URL; a match is skipped, e.g. .(png|jpe?g)$ to skip images. Defaults cover binary files and login/signup pages.
[ "\\.(png|jpe?g|gif|svg|webp|pdf|zip|mp4|css|js)$", "/login", "/signup", "\\?replytocom="]Turn this on to also read /sitemap.xml on each start URL's domain and add its URLs (filtered by the settings above) to the crawl queue, up to Max pages.
Choose what content to put in each row: Markdown only, plain text only, or both. Markdown preserves headings, lists, code blocks and tables.
Optional. Extra CSS selectors to strip before extracting content, e.g. .cookie-banner or #newsletter-signup, on top of the built-in nav/header/footer/aside removal.
[]Enter the maximum number of pages fetched in parallel, e.g. 10. Lower it if the target site rate-limits you.
Turn this on to skip URLs disallowed by the site's robots.txt file (recommended and on by default).
Optional. Select a proxy configuration, e.g. Apify Proxy with the datacenter group, if the target site blocks requests from shared IPs.
{ "useApifyProxy": false}Turn this on to add a chunks array to every page: heading-aware pieces of the Markdown with the heading path and an exact GPT (o200k) token count, ready to embed. Same price — chunks are part of the page result.
Enter the maximum tokens per chunk when Add RAG chunks is on, e.g. 500.
Turn this on to compare every page with the previous run that used the same start URLs and add changeStatus: new, changed or unchanged. Ideal for a scheduled run that keeps a vector DB fresh.
With change detection on, unchanged pages keep their row (url, title, changeStatus) but get empty markdown/text/chunks so you only re-embed what changed. Each page is still one result.
Take the URLs from another Actor run's dataset, e.g. "aBcD1234". Combined with the list above.