Website Content Crawler - Markdown, 45 Fields

Convert any URL into clean LLM-ready markdown. Strips ads, nav, footer. Preserves headings, lists, tables, code blocks. Returns token count. Perfect for RAG, fine-tuning, AI agents. 10x cheaper than Firecrawl.

Convert any webpage into clean Markdown or JSON for AI, RAG, and LLM pipelines. Strips ads, navigation, and cookie banners. Optionally follows links to convert an entire site. Powered by the open-source Crawl4AI library.

URL to Markdown - Clean Web Pages for LLMs & RAG

scrapesage/url-to-markdown

Convert any list of web pages to clean Markdown for LLM/RAG ingestion, notes or docs. Main-content mode (Readability) keeps just the article; full-page mode converts the whole body. Tables and links preserved. No browser. Independent tool, not affiliated with any site it processes.

Turn any public web page or shallow site crawl into clean, LLM-ready Markdown + metadata + optional RAG chunks. Boilerplate/nav/ads removed. Honors robots.txt.

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Turn any web page or site into clean, LLM-ready Markdown and structured JSON for RAG, agents, and fine-tuning. Strips nav/ads/boilerplate; returns main content + metadata.

AST Website Content Crawler for RAG

lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

Advanced web crawler extracting token-chunked Markdown for LLMs & RAG. Features dual-engine hybrid parsing (IBM Docling & Trafilatura) to preserve tables and layouts. Delivers clean text split by native tiktoken counts with rich metadata (headings, levels, tokens) for instant vector search.

Skim Clean Web Reader — URL to Markdown for LLMs & RAG

jessiejanie/skim-clean-reader

Feed your LLM ~4x less input for the same content. Skim turns any URL into clean, LLM-ready Markdown plus metadata in about a second — ads, nav, and boilerplate stripped. Flat $4 per 1,000 pages, no compute billing. Built for AI agents, RAG pipelines, and no-code automations.

Smart Article Extractor - Clean Text & Markdown from Any URL

ninhothedev/smart-article-extractor

$1/1K 🔥 Smart article extractor! Clean title, author, date, text & Markdown from any article URL — perfect for RAG & LLMs. No key. JSON, CSV, Excel or API in seconds. Feed news & blogs into AI in seconds ⚡

Website to Markdown Scraper — LLM & RAG Ready

perforated_hummingbird/url-to-markdown

Scrape any website into clean, LLM-ready Markdown. This URL-to-Markdown converter strips ads, nav, and boilerplate with Mozilla Readability — feed your AI models and RAG pipelines only real content. Batch URLs, optional JavaScript rendering, pay only for pages scraped successfully.

Clean Web Scraper - Markdown for AI via Firecrawl

clearpath/web-to-markdown

Convert any website to clean, LLM-optimized markdown using Firecrawl. Perfect for RAG pipelines, AI training data, and knowledge bases. No login required, 25% cheaper than Firecrawl direct. Batch process hundreds of URLs. Supports PDF/DOCX. Pay only $0.004 per page - no monthly fees.