Fast Website Content Crawler (AI/RAG Markdown, Sitemap & Chunker)

Ultra-fast HTTP/2 website content & sitemap crawler designed for AI agents, Model Context Protocol (MCP) servers, LangChain / LlamaIndex RAG pipelines, and competitive intelligence.

Converts any list of websites, bare domains, or documentation URLs into clean LLM-ready Markdown, token-bounded RAG chunks (ragChunks), structured headings outlines (h1–h6), Schema.org JSON-LD objects, and internal/external link graphs — in seconds, with zero headless browser bloat and zero proxy cost.

Built as a 100% drop-in replacement for 6sigmag/fast-website-content-crawler, 6sigmag/deep-website-content-crawler, and curious_coder/instant-web-scraper.

---

Why Switch to This Fast Website Content Crawler?

Capability	Legacy / Abandoned Crawlers (6sigmag / curious_coder)	Fast Website Content Crawler (This Actor)
LLM / AI Markdown	Raw unformatted text or messy HTML	Clean Markdown (#, ##, -, ```, >) with navbars/ads removed
RAG Readiness	Requires custom tokenizer & splitter	Built-in token-bounded ragChunks (wordCount, estimatedTokens, text)
Subpage Discovery	Homepage only or slow sequential links	Concurrent internal link + /sitemap.xml discovery (useSitemap: true)
Structured Metadata	Title & URL only	JSON-LD (jsonLd), Headings Outline (headings), OpenGraph, Reading Time
Execution Speed	30–120s browser cold start	2–6 seconds via async HTTP/2

---

Input Example

{
  "startUrls": [
    "https://crawlee.dev",
    "apify.com"
  ],
  "maxPagesPerSite": 5,
  "useSitemap": true,
  "includeRagChunks": true,
  "chunkSizeWords": 250
}

---

Output Example

{
  "url": "https://crawlee.dev/",
  "domain": "crawlee.dev",
  "httpStatus": 200,
  "title": "Crawlee · Build reliable crawlers. Fast.",
  "description": "Crawlee is a web scraping and browser automation library for Node.js and Python...",
  "canonicalUrl": "https://crawlee.dev/",
  "language": "en",
  "wordCount": 642,
  "charCount": 4120,
  "estimatedTokens": 854,
  "readingTimeMinutes": 3.21,
  "headingsCount": 12,
  "headings": [
    { "level": "h1", "text": "Build reliable crawlers. Fast." },
    { "level": "h2", "text": "Why Crawlee instead of plain Playwright or Cheerio?" }
  ],
  "markdown": "# Crawlee · Build reliable crawlers. Fast.\n\n## Why Crawlee instead of plain Playwright or Cheerio?...",
  "ragChunksCount": 3,
  "ragChunks": [
    {
      "chunkIndex": 0,
      "wordCount": 250,
      "estimatedTokens": 332,
      "title": "Crawlee · Build reliable crawlers. Fast.",
      "url": "https://crawlee.dev/",
      "text": "# Crawlee · Build reliable crawlers. Fast. ..."
    }
  ],
  "internalLinksCount": 24,
  "externalLinksCount": 8,
  "jsonLd": []
}

---

Explore the Full Zero-Proxy AI/RAG & B2B Lead Suite (@church_of_loujess)

All 7 Actors run on 256 MB RAM with zero external proxies and support 1-click MCP integration for Claude Desktop, Cursor, and n8n:

Actor	Best For
YouTube Transcript, Chapters & AI/RAG Scraper	Android InnerTube captions, chapters, iSummaryMarkdown &
agChunks
YouTube Channel Email, Socials & B2B Leads Scraper	Creator/business emails, socials & external bio-link domain crawling
Google Trends Fast Scraper (No 429)	Interest over time, regional breakdown, rising queries & trending searches
Google News Scraper Fast (Real Decoded URLs + Markdown)	Decodes CBMi.../AU_yqL URLs via Fbv4je RPC + full article Markdown
Website Email, Phone, Socials & Tech Stack Finder	De-obfuscated emails, live DNS/MX mail provider check & 50+ tech signals
Fast Website Content & AI/RAG Markdown Crawler	Parallel sitemap.xml + BFS Markdown crawler & token-bounded
agChunks
Yahoo Finance Stock Quotes, Financials, OHLCV & AI News API	Real-time quotes, OHLCV bars, full Income/Balance/CashFlow & AI tear-sheets

• Open-Source MCP Config, LangChain Loader & n8n Templates: https://github.com/Churchofloujess/apify-zero-proxy-mcp-suite