Universal Webpage to Clean Markdown for AI / LLMs & RAG

Transform any webpage, documentation, news article, or blog post into clean, distraction-free Markdown optimized for Large Language Models (LLMs), AI Agents, LangChain, LlamaIndex, and RAG pipelines.

---

⚡ Why Use This Actor?

Raw HTML pages are bloated with navigation menus, sidebars, cookie banners, tracking scripts, and footer links. Feeding raw HTML into LLMs wastes thousands of expensive prompt tokens and degrades model reasoning.

This Actor solves that problem by:

• 🧹 Stripping Clutter: Uses Mozilla's proven Readability engine to isolate the core readable article and discard boilerplate HTML.
• 📝 Clean Markdown Output: Converts clean HTML into standard GitHub-flavored Markdown (headers, lists, tables, code blocks).
• 💰 Saves LLM Tokens: Includes an option to strip images and estimate token counts, reducing API costs by up to 70%.
• 📊 Structured Metadata: Automatically extracts title, author, publish date, site name, and language.
• 🛡️ Proxy Support: Integrates with Apify Proxy to scrape sites protected by geoblocking or anti-bot protections.

---

🚀 Use Cases

• RAG & Knowledge Base Ingestion: Preprocess URLs into pristine Markdown chunks before embedding into vector databases (Pinecone, Qdrant, Chroma).
• AI Agents & Web Browsing Tools: Equip your LangChain or AutoGen agents with a tool that reads any webpage instantly.
• Dataset Creation: Build high-quality instruction or fine-tuning datasets from documentation and articles.
• LLM Context Injection: Feed clean web research directly into ChatGPT, Claude 3.5, Gemini 1.5, or open-source LLMs without blowing context limits.

---

📥 Input Parameters

Parameter	Type	Required	Default	Description
startUrls	Array	Yes	[]	List of URLs you want to parse into Markdown.
includeMetadata	Boolean	No	true	Extract title, author, publish date, and token estimate.
removeImages	Boolean	No	false	Strip  markdown tags to minimize token size.
proxyConfiguration	Object	No	{ useApifyProxy: true }	Apify Proxy settings for seamless scraping.

---

📤 Output Format

The Actor stores results in Apify Dataset format (exportable as JSON, CSV, Excel, or NDJSON):

{
  "url": "https://example.com/blog/ai-future",
  "metadata": {
    "title": "The Future of AI Agents in 2026",
    "byline": "Jane Doe",
    "excerpt": "A deep dive into how AI agents are reshaping software.",
    "siteName": "Tech Insights",
    "publishedTime": "2026-10-15T08:00:00Z",
    "language": "en",
    "characterCount": 4200,
    "estimatedTokens": 1050
  },
  "markdown": "# The Future of AI Agents in 2026\n\nAI agents are revolutionizing...\n\n- Point 1\n- Point 2\n\n```python\nprint('Hello AI')\n```",
  "scrapedAt": "2026-10-20T08:00:00.000Z"
}

---

🔗 How to Integrate via API

You can call this Actor directly from Python, Node.js, or curl inside your AI application:

Python (LangChain / LlamaIndex / OpenAI)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run = client.actor("YOUR_USERNAME/web-to-clean-markdown-ai").call(
    run_input={
        "startUrls": [{"url": "https://en.wikipedia.org/wiki/Artificial_intelligence"}],
        "removeImages": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["markdown"])