Website to Markdown Crawler for AI & RAG

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Website Content Crawler: Markdown for AI and RAG

automa-flow/website-content-crawler

Turn websites you own or have permission to process into Markdown for AI and RAG. Keep headings, tables and code, with optional chunks linked to each source page. Set a page limit; JavaScript rendering is not supported.

Website to Markdown & RAG Chunk Crawler

groupoject/ai-rag-web-crawler

Crawl public, server-rendered docs and blogs into clean Markdown chunks for RAG, chatbots, and AI agents. One source-labeled row per chunk; no source-site login or AI key.

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

High-performance, lightweight asynchronous crawler converting web pages into clean Markdown, computing token estimates, and generating standard /llms.txt manifests for AI and RAG ingestion.

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

Website to Markdown — Content Crawler for LLM & RAG

webdatatools/website-to-markdown

Content Crawler turns any site into clean Markdown per page for LLMs, RAG pipelines and vector DBs — no headless browser, $1 per 1,000 pages.

Crawl any website and convert every page into clean Markdown for RAG, AI agents and fine-tuning. Main content extraction, sitemap discovery, chunking with overlap and token estimates.

Turn websites and XML sitemaps into clean Markdown for AI, RAG and research. Crawl internal links, keep tables and code, export text and optional chunks. Pay only for extracted pages. No API key required.

Website Content Extractor - Markdown for LLM, RAG, Vector DB

snow_leo_data/website-to-markdown

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.