Website to Markdown Crawler – Clean RAG Content

signal_lab/website-to-markdown-crawler

Website-to-Markdown crawler for RAG and LLM datasets. Extract clean content, SEO metadata, links, JSON-LD, sitemaps, and robots.txt-compliant pages.

News Intelligence Crawler

m_ctim/news-intelligence-crawler

Discover, fetch, parse, and deduplicate publicly available news articles from configurable RSS feeds and Google News search, respecting robots.txt.

Website to Markdown Crawler for LLM & RAG

logiover/website-text-markdown-crawler

Crawl any website to clean Markdown and plain text for LLM training and RAG. HTML to Markdown, no API or login. Export website text to CSV or JSON.

Turn any URL into clean, LLM-ready Markdown for AI agents and RAG pipelines. Enforces robots.txt, extracts main content (Readability) and converts to Markdown. Returns title, byline and markdown.

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

Website Content Crawler Lite

fetch_cat/website-content-crawler-lite

Fast, reliable HTTP website crawler for clean text, Markdown, HTML, metadata, and links. Built for AI/RAG, SEO audits, monitoring, and automation with robots.txt, retries, proxy support, and bounded exports.

Fetch news from RSS feeds and Google News search, then extract clean article text and metadata. Perfect for RAG pipelines, newsletters, trend monitoring, and AI agents. Use via Apify Console/API or connect as an MCP server for Claude, Cursor, and other AI agents.

News & Article Extractor

automation-lab/news-article-extractor

Auto-discover news/blog articles and extract clean text plus Markdown for LLM/RAG corpora. Uses RSS, sitemaps, and Readability; outputs metadata, counts, and token estimates.

Extract clean article text + metadata from any URL: title, author, publish date, full plain text, top image, word count. JSON-LD + Open Graph + readability heuristics, no browser. Use for LLM/RAG ingestion, news monitoring, research agents. Input: url or urls[] (max 1000). Output: JSON.

Smart Article Extractor - Clean Text & Markdown from Any URL

ninhothedev/smart-article-extractor

$1/1K 🔥 Smart article extractor! Clean title, author, date, text & Markdown from any article URL — perfect for RAG & LLMs. No key. JSON, CSV, Excel or API in seconds. Feed news & blogs into AI in seconds ⚡