🚀 AI Web-to-Markdown Crawler

Production-Ready Web Extraction Pipeline for LLMs, RAG Systems & AI Knowledge Bases

Turn any website into clean, structured, AI-ready Markdown — optimized for embeddings, vector databases and Retrieval-Augmented Generation.

Why this Actor?

Most web crawlers collect HTML.

But AI systems don't need HTML.

They need clean, structured, context-preserving knowledge.

This Actor transforms websites into high-quality Markdown documents designed specifically for:

• 🤖 LLM applications
• 🔎 RAG pipelines
• 🧠 AI agents
• 📚 Knowledge bases
• 🗄 Vector databases
• 📖 Documentation search

Instead of feeding your AI:

❌ cookie banners
❌ navigation menus
❌ duplicated content
❌ broken HTML
❌ irrelevant website noise

You get:

✅ clean Markdown
✅ preserved structure
✅ meaningful chunks
✅ metadata-rich documents
✅ embedding-ready output

---

✨ Features

🧠 AI-Optimized Markdown Extraction

Uses a hybrid parsing system combining:

• Trafilatura for fast high-quality article extraction
• Docling for complex documents and structured content

The Actor automatically evaluates extraction quality and selects the best result.

⚡ Adaptive Parsing Intelligence

The crawler learns which parser performs best for each domain.

Example:

example.com → Trafilatura
documentation-site.com → Docling

Future crawls reuse successful strategies automatically.

---

🔥 Built for RAG Pipelines

Every page is converted into semantic chunks optimized for retrieval.

Each chunk includes:

{
  "chunk_id": "...",
  "text": "...",
  "metadata": {
    "title": "...",
    "url": "...",
    "heading_path": [],
    "language": "...",
    "tokens": 512,
    "quality_score": 94.2
  }
}

Perfect for:

• OpenAI embeddings
• Pinecone
• Weaviate
• Qdrant
• ChromaDB
• LangChain
• LlamaIndex

---

🧩 Intelligent Chunking

Unlike simple text splitters, this Actor understands document structure.

Features:

✅ heading-aware splitting
✅ token-based limits
✅ table preservation
✅ code block protection
✅ context overlap
✅ deterministic chunk IDs

Your AI receives meaningful knowledge units instead of random text fragments.

---

🧹 Advanced Content Cleaning

Automatically removes:

• cookie dialogs
• navigation menus
• advertisements
• popups
• scripts
• irrelevant UI elements

Includes quality filtering to reject:

• empty pages
• spam pages
• menu-only content
• duplicate documents

---

🔄 Incremental Crawling

Large documentation websites can be crawled efficiently.

The Actor remembers:

• previously processed content
• failed URLs
• successful parser strategies

Only changed content needs to be processed again.

---

🔐 Production Security

Built with enterprise-style protections:

✅ SSRF protection
✅ private network blocking
✅ cloud metadata protection
✅ URL normalization
✅ content-size limits
✅ retry handling

Safe for automated crawling workflows.

---

📊 Perfect Use Cases

AI Chatbots

Create a chatbot trained on:

• documentation
• help centers
• company websites

Internal Knowledge Bases

Convert:

• company documentation
• manuals
• wikis
• technical pages

into searchable AI knowledge.

Market Research

Extract structured information from:

• competitors
• product pages
• industry websites

Developer Documentation

Create datasets from:

• API docs
• GitHub documentation
• technical references

---

🚀 Example Workflow

Website
   |
   ↓
AI Web-to-Markdown Crawler
   |
   ↓
Clean Markdown
   |
   ↓
Semantic Chunks
   |
   ↓
Embeddings
   |
   ↓
Vector Database
   |
   ↓
AI Assistant / RAG Application

---

Input Example

{
  "startUrls": [
    {
      "url": "https://docs.example.com"
    }
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "parserMode": "auto"
}

---

Output Example

{
  "text": "# Authentication API\n\nThe API allows...",
  "metadata": {
    "source_type": "web_documentation",
    "language": "en",
    "word_count": 1240,
    "reading_time_minutes": 6
  }
}

---

Configuration

Option	Description
maxPages	Maximum pages to crawl
maxDepth	Crawl depth
parserMode	auto / docling / trafilatura
chunkSize	Maximum tokens per chunk
chunkOverlap	Context overlap
respectRobotsTxt	Respect robots.txt
incrementalCrawl	Enable delta crawling

---

Why not use a normal HTML scraper?

Traditional crawlers optimize for:

Collect everything

This Actor optimizes for:

Give AI the right information

The difference matters.

Bad input creates bad AI answers.

Better retrieval starts with better data.

---

Performance Highlights

Designed for real-world production workloads:

• adaptive extraction
• memory-safe document processing
• duplicate detection
• intelligent caching
• scalable crawling architecture

---

Ideal For

⭐ AI developers
⭐ RAG engineers
⭐ SaaS builders
⭐ Data engineers
⭐ Automation specialists
⭐ Companies building AI assistants

---

Try it now 🚀

Turn your website into AI-ready knowledge in minutes.

No custom crawler required.

Just provide a URL and receive structured Markdown optimized for your next AI workflow.