AI & RAG Documentation Ingester (Pre-Chunked Web Crawler)

Stop wasting LLM tokens and vector DB space on website headers, footers, cookie banners, and navigation menus.

This Actor crawls public documentation sites, blogs, and knowledge bases, extracts only the core body content, and outputs clean, pre-chunked text records mapped to their nearest headings—complete with incremental change detection to keep your vector database synced efficiently.

---

🎯 Best For

• RAG & LLM Developers looking to ingest clean documentation, guides, or manuals into vector databases (Pinecone, Qdrant, PGVector, etc.).
• AI Product Teams building custom customer support agents or search engines over vertical/niche websites.
• Knowledge Engineers who need to monitor specific websites and ingest only new or updated pages.

---

Why this is better than a generic crawler

1. Zero Noise: Automatically strips out navigation links, scripts, CSS, sidebars, newsletter boxes, and cookie overlays before parsing.
2. Context-Aware Chunking: Instead of naive character splitting, it generates overlapping text blocks and attaches the relevant heading hierarchy (h1–h6) to every single chunk.
3. Stateful Incremental Ingestion: Uses a persistent Key-Value Store across runs to compare page content hashes. It flags pages as new, changed, or unchanged so you only update changed chunks in your database.

---

💡 Example Workflow: Ingesting a Blog to Pinecone

1. Configure Target: Input the seed URL or sitemap (e.g., https://example.com/sitemap.xml).
2. Filter blog posts: Add https://example.com/blog/** to Include patterns and exclude tags/authors.
3. Enable Chunking & Change Detection: Set chunkText: true and detectChanges: true.
4. Configure Output: Set format to chunks or pagesAndChunks.
5. Sync: Run the Actor, retrieve only the new or changed chunks from the dataset, and upsert them to your vector database.

---

📄 Example Output: Chunk Record

Each chunk is a self-contained record ready for embedding generation:

{
  "recordType": "chunk",
  "chunkId": "a8f9c118bc28a192c73d9059f0f9bde0",
  "pageUrl": "https://example.com/docs/getting-started",
  "canonicalUrl": "https://example.com/docs/getting-started",
  "site": "example.com",
  "title": "Getting Started Guide | Documentation",
  "chunkIndex": 0,
  "chunkText": "To install the library, run 'npm install @sdk/core'. Make sure you have Node.js version 20 or higher installed in your environment before initiating setup...",
  "chunkCharStart": 0,
  "chunkCharEnd": 150,
  "chunkSize": 1000,
  "chunkOverlap": 150,
  "headingsContext": [
    { "level": 1, "text": "Getting Started" },
    { "level": 2, "text": "Installation" }
  ],
  "language": "en",
  "contentHash": "8f3c9e...",
  "timestamp": "2026-06-06T12:00:00.000Z"
}

---

⚙️ Quick Start

1. Start URLs / Sitemap URLs: Provide at least one URL. The default input uses https://example.com/ so the Actor produces a small dataset item without setup.
2. Use Browser Rendering: Toggle on if the page relies heavily on client-side JavaScript (React, Vue, etc.) to render body text.
3. Max Pages Per Site: Bounded limit (default 1) to keep the prefilled run fast and prevent uncontrolled resource use.
4. Chunk Size & Overlap: Match this to your LLM's context window guidelines (e.g., size 1000 chars, overlap 150 chars).

Example Input

{
  "startUrls": [{ "url": "https://example.com/" }],
  "sitemapUrls": [],
  "maxPagesPerSite": 1,
  "includePatterns": [],
  "excludePatterns": [],
  "crawlDepth": 0,
  "maxCrawlRetries": 1,
  "useBrowserRendering": false,
  "languageDetection": true,
  "chunkText": false,
  "chunkSize": 1000,
  "chunkOverlap": 150,
  "outputFormat": "pages",
  "detectChanges": false,
  "storeRawHtml": false,
  "storeCleanText": true
}