Website Content Extractor for LLM and RAG

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.

Input a list of domains. For each, crawl the homepage plus contact/about pages and extract emails, phone numbers, social profile links, physical address, and mailto/tel hrefs. Deterministic (regex + HTML parse), no LLM. One enriched record per domain.

Give it a list of domains. For each, it fetches the homepage and deterministically detects the technology stack from response headers, HTML, scripts, and JSON-LD. No LLM. One record per domain, technologies grouped by category.

AI Website Content Extractor

scrapeai/ai-website-content-extractor

Crawl website pages, strip noise, and convert the main content to clean Markdown for RAG/LLM training.

Website to AI / RAG Data Extractor

scrapyx/website-rag-extractor

Turns any website into a clean, embedding-ready corpus. Strips navigation, footers and cookie banners, converts the real content to markdown, and splits it into overlapping chunks that each carry their own URL, title and metadata. Crawl by URL list, sitemap or link graph.

Turn any article into clean markdown for LLMs. Renders JS-heavy pages, strips ads and navigation, pay only for successful extractions. Best for blog posts, news articles, and documentation pages — not homepage or listing pages.

Scrape posts, pages and content from any WordPress site via the REST API. Clean plain text with authors, categories and tags, ready for LLM training and RAG.

High-performance, lightweight asynchronous crawler converting web pages into clean Markdown, computing token estimates, and generating standard /llms.txt manifests for AI and RAG ingestion.