AI Web Crawler — LLM-Ready Text for RAG Pipelines

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

⚡ Web to LLM Clean Markdown & RAG Crawler

unrivaled_fortress/llm-markdown-clean-web-crawler

Convert any webpage into clean, LLM-ready markdown. Strips ads, navbars, and cookie banners to optimize AI context tokens for ChatGPT, Claude, and RAG.

Public cursor.com/llms.txt headings and links. Not a crawler-readiness score. $0.043 / line.

AI Web Search Scraper - SERP & Page Content for LLM RAG

get_anything/ai-web-search-scraper

Web search API for LLMs: query -> clean results (title, URL, snippet, position, domain) and OPTIONALLY each page's full content as LLM-ready Markdown. Ground ChatGPT/Claude and RAG pipelines with live web results. Google (browser-rendered) or DuckDuckGo engine. Export JSON, CSV or Markdown.

GitHub Repo Scraper - Code & Docs to LLM-Ready Text

get_anything/github-repo-scraper

Chat with a repo: turn any GitHub repository into LLM-ready text - README, file tree and source files as clean text with token counts, plus repo metadata and optional issues. Feed a codebase to ChatGPT/Claude or a RAG pipeline. Official GitHub API, no token required. Export JSON, CSV or Markdown.

Web-to-Markdown Generator for AI & RAG Pipelines

profitstack/web-to-markdown-generator-for-ai-rag-pipelines

Convert any website into clean, heading-based chunking, LLM-ready Markdown for RAG and AI agents.

Website Content Extractor - Markdown for LLM, RAG, Vector DB

snow_leo_data/website-to-markdown

Readable text from 73.1% of 52 live pages over plain HTTP, including 10 of 10 documentation sites. Website to LLM text and a website to markdown converter in one run: clean Markdown, chunks with overlap for a vector database, 46 flat fields, and what changed since the last run.

Turn any URL or website into clean, LLM-ready Markdown or text - boilerplate, ads and nav stripped - with metadata and token count. Feed ChatGPT/Claude context or build RAG pipelines. HTTP-fast with a hardened-browser fallback for JS-heavy pages. Export JSON, CSV or Markdown.

Website to AI / RAG Data Extractor

scrapyx/website-rag-extractor

Turns any website into a clean, embedding-ready corpus. Strips navigation, footers and cookie banners, converts the real content to markdown, and splits it into overlapping chunks that each carry their own URL, title and metadata. Crawl by URL list, sitemap or link graph.

Ai Web Intelligence

hakashi_katake/ai-web-intelligence

AI web crawler and scraper for clean Markdown, text, metadata, JSON-LD, PDFs, RAG chunks, screenshots, and structured web data. HTTP-first crawling with automatic browser fallback, built for AI agents, RAG pipelines, research, and knowledge bases.