Website to Markdown Crawler for LLM & RAG (Sitemaps, JS)

Website to Markdown Crawler for LLM & RAG (sitemaps, JavaScript)

This Actor is a website content crawler that turns a whole site, or just its documentation, blog or help center, into clean Markdown and text, one item per page. It is built to crawl documentation for RAG pipelines, LangChain, LlamaIndex, vector databases (Pinecone, Qdrant, Weaviate) and LLM datasets. Unlike pay-per-usage crawlers, you pay a fixed price per page, with platform usage included.

• Cheap by default. auto mode crawls with fast plain HTTP and renders only client-side (JavaScript) pages with headless Chromium. Static pages are billed at the HTTP price.
• Stays where you point it. Same host and the start URL's path by default, or your own include/exclude globs, plus max depth, max pages and max results.
• Finds more pages. Optional sitemap discovery (robots.txt Sitemap: and /sitemap.xml, including indexes and .gz), /llms.txt and Markdown files.
• Clean content. Mozilla Readability, removal of navigation, footers, cookie banners and dialogs, optional cross-page duplicate-line pruning, canonical-URL de-duplication.
• Extras. Save cleaned HTML (dataset or file), links, screenshots of browser pages, and same-site documents (PDF, DOCX, XLSX…).
• Polite. Always respects robots.txt and Crawl-delay, per-host rate limiting, and treats 401/403/429 as a no rather than something to get around.

Use cases

• Turn product documentation into a knowledge base for a support chatbot.
• Index a blog or help center into a vector database (Pinecone, Qdrant, Weaviate) or a LangChain / LlamaIndex pipeline.
• Build a Markdown corpus of a site for LLM fine-tuning or evaluation.
• Keep an AI assistant up to date by re-crawling docs on a schedule.

Input

Field	Default	What it does
startUrls	–	Where to start. Scope = same host + URLs whose path starts with the start URL's path.
crawlerType	auto	auto (HTTP, browser only when needed), cheerio (HTTP only), playwright:chromium (browser for every page).
includeUrlGlobs / excludeUrlGlobs	–	Glob patterns, e.g. https://example.com/{docs,blog}/**, **/tag/**.
maxCrawlDepth, maxCrawlPages, maxResults	20, 1000, unlimited	Limits (also cap your cost).
useSitemaps, useLlmsTxt	false	Extra URL discovery.
keepUrlFragments, ignoreCanonicalUrl	false	URL identity rules.
htmlTransformer, readableTextCharThreshold	readableTextIfPossible, 100	Main-content extraction.
removeElementsCssSelector, keepElementsCssSelector, removeCookieWarnings, aggressivePrune	defaults	Content cleanup.
dynamicContentWaitSecs, waitForSelector, softWaitForSelector, maxScrollHeightPixels, clickElementsCssSelector, blockMedia	10 s, –, –, 5000, –, true	Browser rendering controls.
saveMarkdown, saveText, saveHtml, saveHtmlAsFile, saveLinks, saveScreenshots, saveFiles	md + text on	What to store.
maxConcurrency, minDelayPerHostMs, requestTimeoutSecs, maxRequestRetries	10, 250, 60, 3	Speed and politeness.
storeSkippedUrls, debugMode, proxyConfiguration	off	Diagnostics and optional datacenter proxy.

{
  "startUrls": [{ "url": "https://docs.apify.com/academy" }],
  "maxCrawlPages": 200,
  "useSitemaps": true,
  "excludeUrlGlobs": [{ "glob": "**/changelog/**" }]
}

Output

{
  "url": "https://crawlee.dev/js/docs/quick-start",
  "crawl": { "loadedUrl": "https://crawlee.dev/js/docs/quick-start", "loadedTime": "2026-11-11T20:30:21.341Z", "referrerUrl": null, "depth": 0, "httpStatusCode": 200, "renderedWith": "raw-http" },
  "metadata": { "title": "Quick Start | Crawlee for JavaScript", "description": "With this short tutorial…", "languageCode": "en", "canonicalUrl": "https://crawlee.dev/js/docs/quick-start", "author": null, "keywords": null, "publishedAt": null, "modifiedAt": null, "openGraph": { "title": "…" }, "jsonLd": [] },
  "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping…",
  "text": "Quick Start\n\nWith this short tutorial…",
  "wordCount": 756,
  "tokenEstimate": 2373
}

Optional fields: html, htmlUrl, links, screenshotUrl. Saved documents appear as { "type": "file", "fileUrl": "…", "contentType": "application/pdf", "sizeBytes": … }. RUN_SUMMARY holds the counts and SKIPPED_URLS the skipped URLs (if enabled).

Pricing

Pay per event, with Apify platform usage included, so there are no compute-unit surprises.

Event	Price	When
Page (HTTP)	$0.10 per 1,000	Each page crawled with plain HTTP and saved
Page (browser)	$0.35 per 1,000	Each page that needed a headless browser for JavaScript
File	$0.50 per 1,000	Each linked document saved (only with saveFiles)

Failed pages, duplicates, robots.txt-blocked and skipped URLs are not charged. Apify adds a tiny per-run start fee. Use maxCrawlPages or Maximum cost per run to cap spending.

Example: a 5,000-page documentation site in auto mode, mostly static, costs about $0.50.

FAQ

How is this different from Website Content Crawler? Similar inputs and output, but a fixed price per page instead of compute units, and a browser only for pages that need it. Static pages cost $0.10 per 1,000 and JavaScript pages $0.35 per 1,000, all usage included.

My site is a React/Vue/Angular app. auto detects client-rendered pages and renders them. If a site is fully client-side, choose playwright:chromium.

Why did it stop at N pages? Check maxCrawlPages, maxCrawlDepth, the path scope (start URL path), robots.txt, and Maximum cost per run.

Can I crawl sites that need a login? No. Cookies and auth headers aren't supported, and the crawler never bypasses logins, paywalls or captchas.

Can I turn off robots.txt? No, it's always respected.

Legal & data use

• Crawls only publicly accessible pages, obeys robots.txt and Crawl-delay, rate-limits per host and never bypasses logins, captchas, paywalls or other access controls.
• No personal-data extraction features (no email or phone harvesting). Output is page content and metadata.
• You're responsible for crawling only sites you're allowed to, and for using the content (copyright, site terms, applicable law), including for AI training or retrieval.

Running on Apify

You can run this Actor on the Apify platform from the Console, the API, a schedule, integrations (webhooks, Slack, Google Sheets, Make, Zapier) or MCP.

Want code? There's a walkthrough with JS and Python examples, plus chunking and embedding: docs site to Markdown for RAG on dev.to.

To use it from Claude, Cursor or another MCP client, add Apify's hosted MCP server and pin this actor: https://mcp.apify.com/?tools=fetch-actor-details,datlab/site-to-markdown-crawler

Other DatLab actors

• Web Page to Markdown — turn a list of URLs into Markdown for LLMs and RAG
• Sitemap Change Monitor — extract sitemap URLs and watch new, removed and updated pages
• ATS Job Listings — Personio, Teamtailor, Workable and Recruitee, one schema
• SAM.gov Contract Opportunities — US federal opportunities from the official daily extract, no API key
• Bulk Website Screenshot — full page or mobile screenshots of a URL list, PNG, JPEG or PDF