⚡ Universal AI Web Markdown & Structured Data Extractor for LLMs & MCP

The Sub-200ms, Token-Dense Alternative to Heavy Headless Crawlers
Engineered specifically for Autonomous AI Agents (Claude Desktop, Cursor, LangChain, CrewAI, LlamaIndex, MCP) and High-Volume Data Pipelines.

---

🚀 Architectural Comparison: Token-Dense AI Extractor vs. Heavyweight Headless Crawlers

Architectural Feature	Traditional Headless Browser Crawlers	Universal AI Markdown & Schema Extractor [MCP]
Extraction Architecture	Full Browser DOM / JavaScript Engine (High Compute & Memory Overhead)	⚡ Async HTTP Stream Parser (Lightweight, Sub-Second Execution)
Output Payload	Raw HTML DOM, Inline CSS, Scripts, Navigation Boilerplate	🎯 Token-Budgeted Clean Markdown (Noise & Ad Elements Stripped)
LLM Context Safeguard	Unbounded Payloads (Prone to Context Window Overflow)	🛡️ Hard-Capped 8,000 Token Ceiling with Truncation Indicator
Structured Schema.org	❌ None (Requires Custom Extraction Rules)	✅ Auto-Extracted JSON-LD (Products, FAQs, Articles, Orgs)
B2B Contact Intelligence	❌ None	✅ Auto-Harvests Emails, Phone Numbers & Social Links
Token Usage Tracking	❌ None	✅ Integrated cl100k_base Token & Character Metrics
Model Context Protocol (MCP)	⚠️ High Timeout & Memory Overhead Risk	✅ Native MCP Support for Claude Desktop, Cursor & Agents
Pricing Model	Variable Compute Unit Charges	💰 Pay-Per-Event: $0.01 / 1,000 Pages ($0.00001/page)

---

🤖 1-Click AI Framework Integrations

1. Claude Desktop & Cursor (Model Context Protocol - MCP)

Add this directly to your claude_desktop_config.json:

{
  "mcpServers": {
    "web-markdown-extractor": {
      "command": "npx",
      "args": ["-y", "@apify/mcp-server", "--token", "YOUR_APIFY_TOKEN"]
    }
  }
}

2. LangChain (Python)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("datalabs/universal-ai-markdown-mcp-extractor").call(
    run_input={"startUrls": [{"url": "https://example.com"}]}
)
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items
clean_markdown = dataset_items[0]["markdown"]
print(f"Extracted {dataset_items[0]['tokenMetrics']['cl100kTokenCount']} tokens.")

3. CrewAI Custom Tool

from crewai.tools import tool
from apify_client import ApifyClient

@tool("Web Content Extractor")
def extract_web_content(url: str) -> str:
    """Extracts ultra-clean markdown, JSON-LD schemas, and contact info from any URL."""
    client = ApifyClient("YOUR_APIFY_TOKEN")
    run = client.actor("datalabs/universal-ai-markdown-mcp-extractor").call(
        run_input={"startUrls": [{"url": url}]}
    )
    items = client.dataset(run["defaultDatasetId"]).list_items().items
    return items[0]["markdown"] if items else "No content extracted."

---

🛠️ Input Parameters

{
  "startUrls": [
    { "url": "https://news.ycombinator.com" },
    { "url": "https://apify.com" }
  ],
  "maxConcurrency": 10,
  "timeoutSecs": 15
}

---

📊 Output Dataset Fields

• url: Final resolved URL (after redirects)
• success: true / false
• statusCode: HTTP response code (e.g. 200)
• markdown: Clean, token-dense Markdown stripped of scripts, navs, footers, and cookie banners
• tokenMetrics: Character count and estimated cl100k token usage
• meta: Title, description, OpenGraph image, canonical URL, language
• jsonLd: Structured Schema.org JSON-LD objects
• contacts: Verified emails, phone numbers, and social URLs (LinkedIn, Twitter, GitHub, etc.)