Web-to-Markdown Generator for AI & RAG Pipelines

The high-precision bridge between the raw web and your LLM. This Actor converts any website into noise-free, chunked Markdown designed specifically for Vector Databases and Retrieval-Augmented Generation (RAG).

Most scrapers return "dirty" data—headers, footers, ads, and navigation menus that waste tokens and dilute the accuracy of your AI. This generator uses smart heuristic logic to strip the boilerplate and deliver only the content that matters.

---

🚀 What it does

• Universal Scraping: Extracts clean text from Wikipedia, blogs, technical documentation (GitBook, Docusaurus), and news sites.
• Smart-Noise-Cancellation: Automatically identifies and removes nav, footer, header, social share buttons, and ad banners.
• Auto-Chunking: Automatically splits long articles into logical blocks based on headers (##), ensuring each piece fits perfectly into an LLM context window.
• Link Sanitization: Converts all relative links into absolute URLs so your AI can always reference the source accurately.

💎 Why it is better

Unlike generic "html-to-markdown" tools, this Generator is purpose-built for AI developers:

1. Token Efficiency: By removing UI junk, you save up to 40% on LLM input tokens.
2. Ready for Indexing: Every output includes a chunkIndex, wordCount, and charCount, making it ready for instant upload to Pinecone, Weaviate, or Milvus.
3. Heuristic Power: It doesn't just look for an <article> tag; it scans for the most likely content container, ensuring success even on non-standard site layouts.

---

💰 Pricing (Pay-Per-Event)

We use a transparent Pay-Per-Event (PPE) model. You only pay for the value you receive—no hidden monthly fees.

Event	Price	Description
Actor Start	$0.001	One-time flat fee to initialize the scraper instance.
Page Scraped	$0.001	Only $1.00 per 1,000 pages successfully processed.

---

📋 How to Run

1. Input URLs: Provide the list of URLs you wish to process in the startUrls field.
2. Configure depth: Set the maxPages limit to control your budget.
3. Proxies: For sites with high bot protection, we recommend using Apify Residential Proxies.
4. Run: Click the Start button. Your data will appear in the Dataset tab in real-time.

---

📊 Clean Output Example

The Actor outputs a flat list of objects. Each row is a perfectly sized "document" ready for your embedding model.

{
  "url": "[https://en.wikipedia.org/wiki/Web_scraping](https://en.wikipedia.org/wiki/Web_scraping)",
  "title": "Web scraping",
  "chunkIndex": 1,
  "markdown": "## History\n\nAfter the birth of the World Wide Web in 1989...",
  "metadata": {
    "wordCount": 176,
    "charCount": 1237,
    "scrapedAt": "2026-01-04T10:00:00.000Z"
  }
}