Sitemap to LLM Markdown & RAG Pipeline

Sitemap to LLM Markdown is an ultra-fast, async Python Actor engineered for AI developers, RAG pipelines, and vector knowledge base ingestion. It crawls XML sitemaps, fetches web pages concurrently, strips ads and navigation boilerplate with a high-performance C-based HTML parser, and outputs clean, token-counted Markdown documents.

---

⚡ Key Features

• Sitemap Index & Gzip Support: Seamlessly processes standard sitemap.xml, multi-tier sitemap_index.xml, and gzip-compressed .xml.gz feeds.
• Aggressive Boilerplate Cleaning: Automatically detects and strips headers, footers, navigation trees, cookie banners, script tags, SVG icons, and advertisements.
• LLM-Ready Formatting: Converts HTML semantic tags (h1-h6, p, pre/code, ul/ol, table, blockquote) into clean GitHub Flavored Markdown.
• Accurate Token Estimation: Estimates LLM token counts and character length for every page, allowing vector embedding chunking on ingestion.
• Pay-Per-Event (PPE) Monetization: Pay only for successful page conversions ($0.001 / page) with zero idle compute waste.
• Ultra-Low Memory Footprint: Runs strictly under 128 MB RAM using async HTTP/2 connections and C-level parsing (selectolax.lexbor).

---

📥 Input Parameters

Parameter	Type	Default	Description
sitemap_urls	Array	["https://apify.com/sitemap.xml"]	Target sitemap URLs to crawl.
max_pages	Integer	10	Maximum number of pages to convert.
include_url_regex	String	None	Optional regex pattern to filter included URLs.
exclude_url_regex	String	None	Optional regex pattern to exclude certain URLs.
concurrency	Integer	5	Concurrent HTTP/2 download connections.
proxy_configuration	Object	None	Optional proxy settings.

---

📤 Output Format

Each record in the default dataset conforms to the following schema:

{
  "url": "https://example.com/docs/getting-started",
  "title": "Getting Started with Python API",
  "description": "Learn how to integrate the API in 5 minutes.",
  "markdown": "# Getting Started with Python API\n\nLearn how to integrate the API in 5 minutes.\n\n## Installation\n\n```bash\npip install mypackage\n```",
  "character_count": 1420,
  "estimated_tokens": 355,
  "extracted_at": "2026-10-18T16:10:00Z"
}

---

💰 Pricing & Monetization

• Startup Event (apify-actor-start): $0.00005
• Per Page Event (page-converted): $0.0010