Docs to RAG Crawler: Site to Markdown

By Johto Data. Crawl a docs site or blog into token-budgeted Markdown, one row per page. Re-crawls charge only new and changed pages.

For AI agents

• Call: run johto.data/site-markdown-crawler with the example input below and read the default dataset.
• Re-crawls: store each row's url and stats.contentHash, pass them back as "knownHashes": {"<url>": "<hash>"}. Unchanged and duplicate pages are skipped and not charged; their links are still followed.
• Context size: stats.tokensEstimate per page; maxTokens trims long pages (stats.truncated flags it).
• Scope: URLs under a start URL, unless you set includeUrlGlobs / excludeUrlGlobs. Cap spend with maxCrawlPages or maxResults.
• Single-page mode: "mode": "single-page" with urls (or urls and no startUrls). One row per URL, same fields as llm-web-reader.
• Safety: http(s) only. Each host is resolved once and refused unless every address is a routable internet address (private, loopback, link-local and metadata addresses are refused); the connection goes only to that checked address, and every redirect hop is re-checked. Refused URLs return a status row at no charge. robots.txt is always respected.

	This Actor	Leading alternative
Crawled page	$0.00025	$0.0005
Single page	$0.0002 (60% under)	$0.0005
Run start	$0.00005	none
Measured run time	10-page crawl: 5.3 s; 1 single page: 4.2 s	not measured

Sample (run mGSasox4jPWgK5bGS, Oct 22 2026, input below):

url	metadata.title	stats.tokensEstimate
https://docs.python.org/3/tutorial/	The Python Tutorial	703
https://docs.python.org/3/tutorial/appetite.html	1. Whetting Your Appetite	1120
https://docs.python.org/3/tutorial/modules.html	6. Modules	6384

Example input (crawls 10 pages)

{
  "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
  "maxCrawlPages": 10,
  "maxCrawlDepth": 1
}

Cost: $0.00255 (10 × $0.00025 + $0.00005).

Output schema

One row per saved page:

• url: final page URL
• crawl: loadedUrl, loadedTime, referrerUrl, depth, httpStatusCode
• metadata: canonicalUrl, title, description, author, keywords, languageCode, publishedAt, url
• text, markdown; html only when saveHtml is true
• stats: markdownChars, tokensEstimate (chars/4), tokensEstimateFull, truncated, contentHash

The run's OUTPUT record holds savedPages, unchangedPagesSkipped, duplicatePagesSkipped, chargedPages.

Single-page mode

{"mode": "single-page", "urls": ["https://docs.python.org/3/tutorial/appetite.html"], "outputFormats": ["markdown"], "maxTokens": 1500}

Per URL: crawl (status, loadedUrl, loadedAt, requestStatus, error on failure), searchResult, metadata, markdown / text / html per outputFormats, stats, links (if includeLinks), unchanged. Failed, robots-blocked and unchanged rows are not charged. Keyword search is not supported: pass page URLs.

Pricing

• apify-actor-start: $0.00005 per run
• page: $0.00025 per saved page (crawl mode)
• single-page: $0.0002 per page read (single-page mode)

Use it when / not when

• Use it for docs sites and blogs that render as plain HTML, for RAG indexes, embeddings or LLM context, and for scheduled re-crawls that should bill only changes.
• Do not use it for pages that need JavaScript, files, screenshots, logins, or for structured records such as prices. Use apify/website-content-crawler for JavaScript sites.

Compatibility

Input startUrls, maxCrawlPages, maxCrawlDepth, includeUrlGlobs, excludeUrlGlobs, useSitemaps, useLlmsTxt, saveMarkdown, saveHtml, htmlTransformer, removeElementsCssSelector, maxResults and output url, crawl, metadata, text, markdown use the same names as Apify's Website Content Crawler.

• MCP: https://mcp.apify.com/?tools=johto.data/site-markdown-crawler
• LangChain / LlamaIndex: ApifyActorsTool("johto.data/site-markdown-crawler")

Limits

Raw HTTP only (no JavaScript rendering). Token counts are estimates. You are responsible for the URLs you supply and for following each site's terms.

Made by Johto Data.