🕸️ Website Content Crawler — Text, Markdown & HTML for AI/LLM

Crawl any website and get clean, structured content from every page — as Markdown, plain text, and HTML. Built to feed LLMs, RAG pipelines, chatbots, and search indexes.

Point it at one or more URLs, set how deep to crawl, and get back the readable content of each page with the navigation, ads, cookie banners, and boilerplate stripped out. No API keys, no glue code.

---

✨ What you get for every page

Field	Description
url / loadedUrl	The page address (after redirects)
title	Page title
description	Meta / OG description
markdown	Clean Markdown of the main content — perfect for LLM ingestion
text	Clean plain-text version
html	Cleaned HTML of the main content block (optional)
canonicalUrl / languageCode	Page metadata
crawl	Depth + referrer URL

---

🎯 Who it's for

• AI / LLM builders — turn a documentation site or knowledge base into Markdown for RAG and fine-tuning.
• Chatbot & support teams — index your help center so an assistant can answer from it.
• SEO & content teams — audit every page's real text content at scale.
• Researchers & data teams — snapshot a whole site into structured records.

---

🚀 How to use it

1. Add one or more Start URLs.
2. Choose a crawler type:
   • Raw HTTP (Cheerio) — fast and cheap, ideal for static or server-rendered sites.
   • Headless browser (Playwright) — renders JavaScript, for single-page apps and JS-heavy sites.
3. Set max pages and crawl depth.
4. (Optional) Narrow the crawl with include / exclude URL patterns.
5. Click Start, then export as JSON, CSV, Excel, or Markdown.

---

📥 Example input

{
  "startUrls": [{ "url": "https://docs.apify.com" }],
  "crawlerType": "cheerio",
  "maxCrawlPages": 50,
  "maxCrawlDepth": 2,
  "stayWithinDomain": true,
  "saveMarkdown": true,
  "saveText": true,
  "saveHtml": false
}

Narrow to just a blog:

{
  "startUrls": [{ "url": "https://example.com" }],
  "includeGlobs": ["https://example.com/blog/**"],
  "excludeGlobs": ["**/tag/**", "**/*.pdf"]
}

---

📤 Example output

One record per page:

{
  "url": "https://docs.apify.com/platform",
  "title": "Apify platform | Apify Documentation",
  "description": "Learn how the Apify platform works.",
  "canonicalUrl": "https://docs.apify.com/platform",
  "languageCode": "en",
  "markdown": "# Apify platform\n\nThe Apify platform helps you...",
  "text": "Apify platform\nThe Apify platform helps you...",
  "textLength": 1843,
  "crawl": { "depth": 1, "referrerUrl": "https://docs.apify.com", "loadedUrl": "https://docs.apify.com/platform" }
}

Export in one click to JSON, CSV, Excel, or Markdown, or pull results via the Apify API.

---

🔌 Integrations & scheduling

• Schedule the crawler to keep your knowledge base / index fresh.
• Pipe the Markdown straight into a vector database (Pinecone, Weaviate, pgvector) for RAG.
• Combine with LangChain / LlamaIndex — the Markdown output drops right into a document loader.

---

❓ FAQ

Cheerio or Playwright — which do I pick? Start with Cheerio (fast, cheap). If pages come back empty or missing content, the site renders with JavaScript — switch to Playwright.

How does it decide what's "main content"? It removes nav, header, footer, sidebars, ads, and cookie banners, then extracts the primary <main> / <article> region (falling back to the body). You get the article, not the chrome.

Will it stay on my site? By default it only follows links on the same domain. Use include/exclude globs for finer control.

Do I need a proxy? Apify Proxy is on by default to reduce blocking. For open public sites you can turn it off to save cost.

Is crawling legal? It fetches publicly available pages. You're responsible for respecting each site's terms and robots policies and for how you use the content.

---

🗺️ Roadmap

• robots.txt and sitemap.xml support
• Readability-grade main-content scoring
• Per-page screenshots (Playwright mode)
• Automatic language detection & chunking for RAG

---

Built and maintained by hichemdev. Found a bug or want a feature? Open an issue on the Actor's Issues tab.