Web to LLM Markdown & RAG Clean Extractor

🚀 Convert any website, documentation hub, or blog into clean, token-optimized Markdown ready for LLMs, Claude, GPT-4, Cursor, and RAG pipelines.

---

💡 Why this Actor?

When scraping the modern web for AI context windows or Vector DBs (Pinecone, Qdrant, Chroma), standard scrapers dump bloated HTML full of:

• Cookie consent banners & popups
• Navbars, footers & sidebar links
• Tracking scripts, styles & SVG icons
• Ads, promotional banners & social share widgets

This boilerplate wastes up to 70% of your LLM context tokens and severely degrades model reasoning.

Web to LLM Markdown Extractor solves this by:

1. Intelligently identifying main content (<article>, <main>, #content, .docs-content, etc.).
2. Aggressively stripping boilerplate (cookie banners, scripts, navigation, ads).
3. Converting to clean, semantic Markdown (ATX headings, fenced code blocks, bullet points).
4. Calculating token & word metrics so you know exact context costs upfront.
5. Lightweight & Blazing Fast: Pure HTTP crawler powered by Crawlee & Cheerio. No heavy browser overhead, uses minimal RAM (128–256 MB), and crawls hundreds of pages in seconds.

---

⚙️ Features

• 📑 Single URL or Full Domain Crawling: Scrape a single article or crawl an entire documentation site with maxDepth control.
• 🎯 Domain-Bound Crawling: Stays on the same hostname so you don't leak into external sites.
• 🧹 Deep Noise Filtering: Automatically drops cookies, footers, headers, ads, and inline styles.
• ⚡ Token Saver Mode (stripImages): Strip ![alt](url) image tags to minimize LLM token usage.
• 🔗 Clean Text Mode (stripLinks): Convert hyperlinks into plain text for pure text embedding.
• 📊 Token & Word Metrics: Provides instant tokenEstimate and wordCount for every page.

---

📥 Input Configuration

Parameter	Type	Default	Description
startUrls	Array	["https://docs.github.com/en/get-started"]	Starting URLs or docs hubs to process.
maxPages	Integer	10	Maximum number of pages to crawl and convert.
maxDepth	Integer	1	Crawl depth: 0 = only given URLs, 1 = follow internal links 1 level deep.
stripImages	Boolean	true	Remove image tags to save prompt context tokens.
stripLinks	Boolean	false	Convert hyperlinks to plain text.

Example Input:

{
  "startUrls": [
    { "url": "https://docs.github.com/en/get-started" }
  ],
  "maxPages": 15,
  "maxDepth": 1,
  "stripImages": true,
  "stripLinks": false
}

---

📤 Output Dataset Format

Each record in the Apify dataset contains clean metadata and pure Markdown:

{
  "url": "https://docs.github.com/en/get-started",
  "title": "Get started with GitHub documentation",
  "description": "Learn how to use GitHub with step-by-step guides and documentation.",
  "author": null,
  "language": "en",
  "markdown": "# Get started with GitHub documentation\n\nGitHub is a code hosting platform for version control and collaboration...\n\n## Quickstart\n\n1. Sign up for GitHub\n2. Create a repository\n3. Start collaborating\n",
  "wordCount": 384,
  "tokenEstimate": 450,
  "crawledAt": "2026-09-17T15:10:00.000Z"
}

---

🛠️ Python & LangChain Quickstart

Use the Apify Python client to ingest web docs directly into your RAG pipeline:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

# Run the Actor
run = client.actor("vujofix/web-to-markdown-llm").call(run_input={
    "startUrls": [{"url": "https://docs.stripe.com/api"}],
    "maxPages": 20,
    "maxDepth": 1,
    "stripImages": True
})

# Fetch cleaned Markdown records
dataset = client.dataset(run["defaultDatasetId"]).list_items().items
for page in dataset:
    print(f"Title: {page['title']}")
    print(f"Estimated Tokens: {page['tokenEstimate']}")
    # Pass page['markdown'] to your Vector Store or LLM context

---

💰 Pricing

• Pay-Per-Event: $2.00 / 1,000 pages
• Incredibly cost-effective: A full documentation scrape of 100 pages costs just $0.20, saving hours of manual cleanup and hundreds of dollars in wasted LLM tokens.

---

🛡️ License

Apache-2.0