AI Website to Clean Markdown & JSON Scraper (RAG Ready)

Convert noisy, bloated web pages into clean, structured Markdown (.md), JSON, and context-aware RAG chunks in seconds. Built specifically for AI Agents, LLM pipelines, and Vector Databases (Pinecone, Qdrant, Weaviate, Milvus).

By stripping out HTML boilerplate, navigation menus, ads, inline scripts, and CSS styling, this Actor reduces your raw webpage token footprint by 80% to 92%+—slashing your OpenAI, Anthropic, and Gemini API costs while improving retrieval accuracy.

---

Key Features for AI Developers

• Hybrid Scraping Engine (Auto Mode): Uses ultra-fast HTTP/2 requests (httpx) first for low compute cost and automatically falls back to a headless Chromium browser (Playwright) only when JavaScript/SPA rendering (React, Next.js, Vue) is required.
• Massive LLM Token Savings: Calculates and reports exact token reduction metrics (raw_html_tokens vs. clean_output_tokens and tokens_saved_percentage) on every scraped URL.
• Built-In RAG Auto-Chunking: Automatically splits extracted Markdown into semantic, overlapping character/token chunks so you can push embeddings directly into a Vector DB without extra preprocessing.
• Rich Metadata Extraction: Automatically extracts page title, author, date, sitename, description, categories, and tags.
• Table & Link Preservation: Converts complex HTML tables into clean GitHub-Flavored Markdown tables and preserves inline reference links.
• Anti-Bot & Proxy Support: Native integration with Apify Datacenter and Residential proxies.

---

Input Parameters

Parameter	Type	Default	Description
startUrls	Array	Required	List of website URLs to scrape and convert.
scrapingMode	String	"auto"	"auto" (HTTP + Playwright fallback), "fast_http" (fastest/cheapest), or "playwright_browser" (for JS-heavy SPAs).
outputFormat	String	"markdown"	Choose "markdown", "json", or "both".
includeLinks	Boolean	true	Preserve hyperlinks inside extracted Markdown.
includeImages	Boolean	false	Include image URLs in Markdown output.
includeTables	Boolean	true	Convert HTML tables into clean Markdown tables.
enableChunking	Boolean	true	Split clean content into overlapping chunks for RAG & Vector DBs.
chunkSize	Integer	1500	Maximum character length per RAG chunk (~4 characters = 1 token).
chunkOverlap	Integer	200	Character overlap between consecutive chunks to preserve context.
maxConcurrency	Integer	5	Number of URLs to process in parallel.
proxyConfiguration	Object	Apify Proxy	Configure Apify Proxy to bypass rate limits and anti-bot protections.

---

Sample Output

Below is a real extraction result from https://en.wikipedia.org/wiki/Retrieval-augmented_generation demonstrating a 92.2% reduction in LLM tokens:

{
  "status": "success",
  "url": "[https://en.wikipedia.org/wiki/Retrieval-augmented_generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation)",
  "engine_used": "httpx",
  "title": "Retrieval-augmented generation",
  "metadata": {
    "title": "Retrieval-augmented generation",
    "author": "Contributors to Wikimedia projects",
    "date": "2026-10-01",
    "sitename": "Wikipedia",
    "description": "Technique for enhancing large language model accuracy with external data",
    "categories": [],
    "tags": []
  },
  "token_stats": {
    "raw_html_tokens": 68450,
    "clean_output_tokens": 5340,
    "tokens_saved": 63110,
    "tokens_saved_percentage": "92.2%"
  },
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information...",
  "structured_json": null,
  "rag_chunks_count": 19,
  "rag_chunks": [
    {
      "chunk_id": 0,
      "char_length": 1184,
      "estimated_tokens": 296,
      "text": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information..."
    }
  ],
  "scraped_at": "2026-10-22T03:55:00.000000+00:00"
}