Website Content Crawler Pro — Clean Markdown for AI, RAG & LLMs

Turn any website into clean, LLM-ready Markdown (or text) in one run. Built for feeding documentation, blogs and knowledge bases into RAG pipelines, vector databases and AI agents. Crawls with a fast HTTP engine or a full JavaScript browser, extracts just the main content (no nav, ads or cookie banners), and follows internal links with precise glob and sitemap control. No account, no API key.

Features

• 📝 Markdown output — clean, structured Markdown per page, ideal for LLMs, RAG and vector DBs (plus plain text and optional HTML)
• 🌐 HTTP or Browser mode — fast HTTP crawling for static sites, or full Chromium rendering for JavaScript/SPA sites
• 🧹 Main-content extraction — Readability strips navigation, ads, footers and cookie banners, keeping the real article
• 🗺️ Sitemap + glob crawling — seed from sitemap.xml, and include/exclude URLs with glob patterns
• 🔗 Smart link following — configurable depth, same-domain-only, and total page cap
• 🎯 CSS control — keep-only or remove specific elements; wait for selectors and dynamic content in browser mode
• ⚡ Fast & parallel — tunable concurrency, retries and proxy support

What you get

One record per page:

{
  "url": "https://docs.example.com/guide",
  "title": "Getting Started",
  "description": "A short guide to …",
  "author": null,
  "siteName": "Example Docs",
  "lang": "en",
  "wordCount": 812,
  "markdown": "# Getting Started\n\nThis guide walks you through …",
  "text": "Getting Started This guide walks you through …"
}

Input

{
  "urls": ["https://docs.apify.com/academy"],
  "crawlerType": "http",
  "crawl": true,
  "maxCrawlPages": 200,
  "includeUrlGlobs": ["https://docs.apify.com/academy/**"],
  "saveMarkdown": true
}

Field	Description
urls	Start URLs to crawl.
crawlerType	http (fast) or browser (JS rendering).
crawl	Follow internal links from the start URLs.
maxCrawlPages / maxCrawlDepth	Total page cap and link-hop depth.
includeUrlGlobs / excludeUrlGlobs	Glob filters for which URLs to crawl.
useSitemaps	Seed from each site's sitemap.xml.
saveMarkdown / saveText / saveHtml	Output formats.
htmlTransformer	readability (main content) or none (full body).

Use cases

• RAG & AI knowledge bases — turn docs sites into Markdown for retrieval-augmented generation
• LLM fine-tuning datasets — collect clean article text at scale
• Docs & blog migration — export a site's content to Markdown
• Competitive/content research — pull structured page content across a domain

Pricing

Pay-per-event: a small amount per page crawled. See the Pricing tab for current rates.

FAQ

What's the difference from a basic content scraper? This Pro version outputs clean Markdown (not just text), renders JavaScript sites with a real browser, extracts the main content with Readability, and supports sitemaps + glob filters — everything a RAG/LLM pipeline needs.

Do I need an API key? No. Just give it URLs and run — no account or key required.

Can it handle JavaScript / single-page apps? Yes. Set crawlerType to browser and, if needed, waitForSelector or dynamicContentWaitSecs so client-rendered content loads before extraction.

What output format is best for AI/RAG? Markdown. It preserves headings, lists and links in a compact form that LLMs and chunkers handle well. Plain text and HTML are also available.

How many pages can it crawl? Up to tens of thousands per run via maxCrawlPages, with sitemap seeding and glob filters to target exactly the pages you want.

Related Actors

• Website Content Crawler — the lightweight text-only crawler
• Website Contact Scraper — emails, phones and socials from any site
• RAG Web Browser — search the web and return page content for AI agents
• Bulk URL Checker — status, redirects and metadata for a list of URLs

Notes

Original clean-room implementation using open-source libraries (Crawlee, Mozilla Readability, Turndown). Respects the target site's content; use responsibly and within each site's terms. This is an independent tool, not affiliated with any site it crawls.