Website to RAG Chunks — RAG crawler, token-sized, cited

RAG Web Browser

apify/rag-web-browser

Web search and fetch tool for AI agents and RAG pipelines. It queries Google Search, scrapes the top N pages using a full web browser, and returns their content as clean Markdown for further processing by an LLM. Can also fetch individual URLs.

Website Content Crawler - Clean Text, Markdown & RAG Chunks

topapi/website-content-crawler

Crawl any website or URL list for $1.20 per 1,000 pages: one row per page with clean text, Markdown, HTML, metadata, links, headings, tables and RAG chunks. Plain HTTP first, headless Chrome only for JavaScript pages. Sitemaps, llms.txt, PDF/DOCX text, Bing/DuckDuckGo search, change monitor.

Website & Page to Markdown for RAG and LLMs

humble-echidna/page-to-markdown

Turn web pages or whole sites into clean Markdown for RAG and LLMs: main content only, with title, links and word count. Whole-site mode reads the sitemap first, can return only changed pages, and splits pages into heading-based chunks. Respects robots.txt and AI opt-outs. No browser.

Rag Knowledge Graph Builder

cspnair/rag-knowledge-graph-builder

Transform websites into RAG-ready datasets. Crawls pages, chunks content into semantic segments (500-1000 tokens), and generates hypothetical questions for each chunk. No API key needed with native mode. Output: pre-indexed JSON optimized for AI retrieval with 3x better accuracy than raw text.

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Website to Markdown Crawler | Pre-Chunked for RAG

themineworks/rag-crawler

Crawl any website into clean, pre-chunked Markdown with per-chunk token counts for RAG pipelines, vector DBs (Pinecone, Qdrant) and LLM context. MCP native for Claude & ChatGPT. SPA support via Playwright. Pay only for pages that crawl. A Firecrawl alternative.

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Google RAG Browser

visita/rag-browser

This Actor provides essential web browsing and content extraction functionality for AI Agents, LLM applications, and Retrieval-Augmented Generation (RAG) pipelines. It functions similarly to the web search feature in popular LLM chatbots, providing fresh, contextualized data directly from the web.

AST Website Content Crawler for RAG

lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

Advanced web crawler extracting token-chunked Markdown for LLMs & RAG. Features dual-engine hybrid parsing (IBM Docling & Trafilatura) to preserve tables and layouts. Delivers clean text split by native tiktoken counts with rich metadata (headings, levels, tokens) for instant vector search.

Markdown RAG Chunker: Document Text Splitter

codepoetry/markdown-rag-chunker

RAG chunking for PDFs, Word, PowerPoint, Excel, EPUB and web pages. A header-aware text splitter keeps each chunk inside one section, caps it at 512 tokens by default, and adds its header path, source, stable ID and exact token count. Ready to embed into Pinecone, Qdrant or pgvector.