🤖 AI-Ready Website to Clean Markdown, Token Counter & RAG Chunker

Stop feeding noisy HTML to ChatGPT, Claude, and your Vector Databases. Most web pages contain 85% to 96% HTML noise (navigation menus, cookie banners, scripts, SVG icons, and tracking bloat) that burns your OpenAI/Anthropic API budget and degrades RAG retrieval accuracy.

AI-Ready Website to Clean Markdown, Token Counter & RAG Chunker strips out 100% of DOM noise in 1.5 seconds per page using ultra-lightweight HTTP/2 requests—delivering clean hierarchical Markdown, context-injected RAG chunks, auto-extracted Q&A pairs, and a Multilingual Token Savings Audit (EN, ES, PT, FR, DE, IT).

---

🔥 Why AI Engineers & n8n/Make Builders Choose This Actor

Feature	Standard Markdown Scrapers	AI-Ready Markdown & RAG Chunker (duartelabs)
Execution Speed & RAM	Heavy headless browsers (30–60s, high compute cost)	1.5 seconds via pure HTTP/2 (~$0.00 compute)
Smart RAG Chunking	Dumps one massive text string that breaks embeddings	Built-in Semantic Chunker (smartChunks) with Parent Context Headers
LLM Token & Cost Audit	None	Calculates Raw vs. Clean Tokens, % Saved & Est. GPT-4o Cost
Auto Q&A Extraction	None	Automatically extracts Question/Answer pairs (extractedFAQs) for Chatbot training
Markdown Tables & Links	Often broken or stripped	Converts HTML tables to clean Markdown tables & resolves absolute URLs
Multilingual Audit Summary	English only	6 Languages (EN, ES, PT, FR, DE, IT)

---

📦 Sample Output JSON

{
  "url": "https://apify.com/about",
  "domain": "apify.com",
  "title": "About Apify - Full-stack web scraping and data extraction platform",
  "detectedLanguage": "en",
  "rawHtmlTokens": 38420,
  "cleanMarkdownTokens": 1240,
  "tokenSavingsPct": 96.8,
  "estimatedGpt4oCostUsd": 0.0031,
  "wordCount": 935,
  "readingTimeMinutes": 4.7,
  "totalChunks": 3,
  "aiReadinessSummary": "✅ [AI & RAG READY] apify.com: Stripped 96.8% of noisy HTML bloat (38,420 → 1,240 LLM tokens). Structured into 3 context-aware vector chunks and 2 extracted Q&A pairs.",
  "smartChunks": [
    {
      "chunkIndex": 1,
      "sectionHeading": "What we do",
      "tokenEstimate": 420,
      "charLength": 1680,
      "ragText": "# About Apify > What we do\nSource: https://apify.com/about\n\nApify is a full-stack web scraping and browser automation platform..."
    }
  ],
  "extractedFAQs": [
    {
      "question": "Why did we launch Apify?",
      "answer": "We built Apify to make the web more programmable and help developers extract structured data effortlessly."
    }
  ]
}

---

🔗 Seamless Integration with n8n, Make, LangChain & Vector DBs

1. Pinecone / Qdrant / Supabase pgvector: Map smartChunks[].ragText directly into your OpenAI text-embedding-3-small node—no custom text-splitting code needed!
2. Custom GPTs & Claude Projects: Feed cleanMarkdown directly into your LLM prompt while saving 90%+ on input token costs.
3. AI Support Chatbots: Use extractedFAQs to populate instant FAQ knowledge bases.

---

🤖 1-Click Integration with Claude Desktop & Cursor (MCP)

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": ["-y", "@apify/actors-mcp-server", "--actors", "duartelabs/ai-website-to-markdown-rag-chunker"],
      "env": { "APIFY_TOKEN": "YOUR_APIFY_API_TOKEN" }
    }
  }
}

---

🛠️ Complete Your Automation Stack by duartelabs

• 🟢 B2B Lead, WhatsApp, Email & Ad-Pixel Intelligence Scraper: Extract verified WhatsApp numbers, decision-maker emails, 9+ Ad Pixels, Schema.org data, and multilingual outreach hooks.
• 🟠 Shopify Competitor Price, Stock, Discount & Best-Sellers Spy: Extract complete competitor product catalogs, live discounts, stock availability, and AI pricing insights in seconds.
• 🔵 Technical SEO, Meta, Schema.org & Broken Links Auditor: Audit websites for 0–100 Technical SEO Score, missing Meta/H1 tags, Schema.org JSON-LD, image ALT gaps, response speed, and broken 404 links in 6 languages.

---

⚡ Ready-to-Import n8n Workflow Templates & Claude/Cursor MCP Config (1-Click Download)

Import these pre-built JSON workflows directly into n8n (Workflows -> Import from URL / File) or Claude Desktop / Cursor IDE:

• 🤖 Complete 4-Actor Claude Desktop & Cursor MCP Config (claude_cursor_mcp_config.json)
• 🟢 n8n B2B Lead, WhatsApp & Ad-Pixel Enricher Template (n8n_b2b_lead_whatsapp_workflow.json)
• 🟠 n8n Shopify Competitor Price & Stock Spy Template (n8n_shopify_competitor_spy_workflow.json)
• 🟣 n8n AI Website to Clean Markdown & RAG Chunker Template (n8n_ai_markdown_rag_chunker_workflow.json)
• 🔵 n8n Technical SEO, Schema & Broken Links Auditor Template (n8n_technical_seo_auditor_workflow.json)