🤖 HelpDesk-to-RAG Markdown Scraper

Transform messy knowledge bases and help centers into perfectly clean, structured Markdown—ready for instant ingestion by Large Language Models (LLMs) and RAG (Retrieval-Augmented Generation) pipelines.

🚀 Why Use This Scraper?

When building custom AI agents or AI customer support bots, feeding them raw HTML confuses the models with navigation bars, tracking scripts, and footer links. This wastes your API tokens and degrades the AI's answers.

This Actor solves that problem. It uses Mozilla's Readability Engine to isolate the core article content and Turndown to convert it into pristine Markdown.

• Lower Token Costs: Stripping out heavy HTML tags saves massive amounts of tokens when passing context to OpenAI, Anthropic, or local LLMs.
• Better Vector Search: Clean Markdown headers (#, ##) allow semantic chunking tools (like LangChain or LlamaIndex) to split your documents intelligently.
• Zero Maintenance: It automatically handles internal link discovery, proxy rotation, and pagination across domains.

⚙️ Core Features

• Deep Crawling: Provide a single starting URL (e.g., https://docs.apify.com), and the Actor will automatically find and queue all other articles on that same domain.
• Smart Content Extraction: Automatically identifies and extracts only the main article, ignoring sidebars, ads, and footers.
• Anti-Blocking Built-In: Fully integrated with Apify's Proxy network to bypass rate limits and IP bans without any configuration required on your end.

📝 Input Configuration

This Actor is designed to be simple to use. Here are the available inputs:

Field	Type	Description	Default
Start URLs	Array	The URLs of the help centers or specific articles you want to scrape.	[{"url": "https://docs.apify.com/"}]
Max Pages	Integer	The absolute maximum number of pages to crawl before the Actor stops.	50
Remove Boilerplate	Boolean	Strips navbars, ads, and footers before converting to Markdown. Highly recommended for AI use cases.	true

💰 Cost and Pricing

This Actor operates on a highly efficient Pay-per-event (PPE) model. You only pay for what you actually get! Price: $4.00 per 1,000 results ($0.004 per successfully extracted Markdown article). Platform Costs: Free / Included! The Apify platform computing costs are fully baked into the per-article price. You are never charged for failed page loads, redirects, or blocked requests.

🔗 Seamless Integrations

Because this Actor outputs clean JSON and Markdown, it can be easily connected to your favorite AI tools using the Apify API or Webhooks: Vector Databases: Pinecone, Milvus, Weaviate, Qdrant AI Frameworks: LangChain, LlamaIndex, Flowise Automation: Zapier, Make.com Built with ❤️ for the AI developer community.

📊 Output Data Format

The Actor stores its results in the Apify Dataset. You can instantly download your data in JSON, CSV, XML, or Excel.

Here is an example of a single extracted article in JSON format:

{
  "url": "[https://docs.apify.com/academy/getting-started](https://docs.apify.com/academy/getting-started)",
  "title": "Getting Started | Apify Documentation",
  "markdown": "# Getting Started\n\nWelcome to the Apify platform! In this guide, you will learn how to build your first web scraper...\n\n## Prerequisites\n* A free Apify account.\n* Basic knowledge of JavaScript.",
  "scrapedAt": "2026-07-28T12:00:00.000Z"
}