🚀 Smart RAG Optimizer

Smart RAG Optimizer is an enterprise-grade, high-performance web crawler and text preprocessor designed specifically to convert any web page into clean, chunked, RAG-ready JSON data for large language Models (LLMs) and Vector Databases.

---

✨ Key features

• 🧹 Automated Noise Removal: Automatically strips out non-content elements such as scripts, styles, navigation bars, footer, headers, ads, and popups.
• 🌐 Automatic Sitemap Detection: Automatically detects and extracts links from sitemap.xml as a primary entry point for full-domain coverage.
• ⚡️ Hybrid Engine (Cheerio + Playwright): Uses CheerioCrawler for ultra-fast parsing, with seamless optional fallback to PlaywrightCrawler (useBrowser) to bypass firewalls and JavaScript-rendered pages.
• 📊 Markdown Table Conversion: Automatically transforms complex HTML tables into clean Markdown format to preserve structured context for LLMs.
• ✂️ Smart Semantic Chunking: Dynamic paragraph-aware splitting with customizable overlap (chunkOverlap) to eliminate context boundary truncation.
• 🎯 Rich RAG Metadata: Emits Structured JSON complete with chunkId, estimated tokensCount, headingPath breadcrumbs, and raw cleaned text.
• 🛡️ Built-in Proxy Integration: Direct integration with Apify proxy for dynamic IP rotation and anti-bot mitigation.

---

💡 Common Use Cases

• Preparing web data for RAG pipelines (LangChain, LlamaIndex, OpenAI Assistant API, AutoGen).
• Feeding structured context to Vector Databases (Pinecone, Weaviate, Qdrant, Chroma, Milvus).
• Building high-accuracy custom Knowledge Bases for Enterprise AI Chatbots.

---

🛠️ Input Parameters

Parameter	Type	Default	Description
url	string	(Required)	The target webpage URL to extract and process.
chunkSize	number	1000	Target character count threshold per chunk.
chunkOverlap	number	200	Character overlap between consecutive chunks to retain semantic context.
maxPages	number	1	Maximum number of pages to crawl starting from the URL or Sitemap.
chunkByHeaders	boolean	true	Enable smart semantic chunking based on natural paragraphs and sections.
useBrowser	boolean	false	Enable Playwright headful browser rendering to handle heavy JS and bypass firewalls.

---

📄 Output Sample

[
{
    "url": "https://docs.python.org/3.11/",
    "title": "Python 3.11.18 documentation",
    "totalChunks": 3,
    "chunks": [
    {
    "chunkIndex": 1,
    "chunkId": "chunk_1_aHR0cHM6Ly",
    "tokensCount": 475,
    "headingPath": "Python 3.11.18 documentation > Part 1",
    "content": "Welcome! this is the official documentation for python 3.11.18..."
    }
    ],
    "rawText": "Welcome! this is the official documentation for python 3.11.18..."
}
]