Site to LLM Knowledge Base

Crawl any website and extract clean Markdown optimized for LLM training, RAG pipelines, and AI knowledge bases - removes boilerplate and outputs structured JSON with URL, title, markdown, and metadata.

Website Content Crawler API - Markdown for RAG

tugelbay/website-content-crawler

Crawl public websites and extract clean Markdown, text, or HTML for RAG pipelines, AI agents, documentation indexing, and content monitoring. Guide: https://konabayev.com/tools/website-content-crawler/?utm_source=apify_info&utm_medium=referral&utm_campaign=website-content-crawler

Web Scraper For Llms

abotapi/web-scraper-for-llms

Stealth web scraping engine built for LLMs. Converts any web page to clean markdown or HTML

Extract Schema.org JSON-LD structured data from any website. Fast, lightweight HTTP-based scraper that pulls all JSON-LD scripts - perfect for SEO analysis, product data extraction, and AI/RAG pipelines. No browser overhead.

Web Page to Markdown Converter Pro — Clean Content Extraction

maged120/url-to-markdown-pro

Convert any web page to clean, formatted Markdown. Advanced content extraction handles complex layouts, paywalls, and JavaScript-rendered pages — strips ads and returns just the content.

Crawl any website and extract clean Markdown and plain text optimized for AI ingestion, RAG pipelines, and LLM context. Readability-style main content extraction removes ads, navs, and footers. Configurable depth, concurrency, and page limits. Pay-per-page.

AI Dataset Converter - Website to Training Data

boztek-ltd/ai-dataset-converter

Crawl websites and convert content into AI-ready formats: RAG chunks, fine-tuning JSONL, Q&A pairs, clean Markdown. Token-aware chunking, quality scoring, deduplication. No external LLM API needed.

Pre-chunked markdown website crawler for RAG pipelines and LLM context. Only charges for pages that successfully crawl.

Second Brain Builder — Karpathy's LLM Wiki, Automated

openclawai/second-brain-builder

Transform web content into an organized Obsidian-compatible knowledge base with AI-powered compilation. Scrapes any website and generates a Second Brain structure with wikilinks.

Content Intelligence Extractor

werednalux/content-intelligence-extractor

Extract clean Markdown from Reddit threads and news sites. Built for LLM pipelines, n8n workflows, and AI content analysis. Uses Mozilla Readability + Reddit JSON API for noise-free output.

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.