Website to Markdown for AI - Fast, Cheap, RAG-Ready Crawler

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

Website to RAG Chunks: Vector-DB Ready Markdown

torix33/website-to-rag-chunks

Crawl a website, docs portal or blog and get LLM-ready chunks for RAG: clean Markdown split by headings to your token budget, with overlap, heading path, chunk IDs, content hashes and page metadata. Drop straight into Pinecone, Qdrant, Weaviate, pgvector or LangChain. Pay per page.

Extract hiring signals from company career pages, including role titles, departments, remote work language, locations, and applicant-tracking links.

Crawl a docs site, help center or blog and get embedding-ready chunks: main content as Markdown, split at headings to a token size you choose, each with its heading path, a citation URL to its section, a token count and a content hash. Plain HTTP. Built and maintained by an AI agent.

Website to Markdown Crawler (llms.txt)

sfoln/website-to-markdown

Crawl a website or docs section into clean, LLM-ready markdown: sitemap first, then links, one row per page with title, depth, word and token counts, plus llms.txt and llms-full.txt files for AI agents and RAG. Polite, no browser needed. $1.50 per 1,000 pages.

Website Content Crawler to Markdown for AI & LLMs

deepmine/website-content-crawler

Website content crawler for AI, RAG and LLMs: give start URLs or a sitemap, get every page's clean Markdown and text with title, description, headings, author, dates, canonical URL and links. Fast HTTP crawling, URL globs, depth and page limits, sitemaps, llms.txt and PDFs. No login.

Extract the text of PDF files and scanned images, one row per page. Digital pages are read from their text layer at $0.50 per 1,000 pages, scans go through Tesseract OCR at $4 per 1,000 pages in 7 languages. Failed and blank pages are free.

Deterministic European Accessibility Act (EAA) & WCAG 2.2 AA accessibility auditor for websites and landing pages. Extracts concrete CSS selector violations and ready-to-merge remediation patches. MCP ready.

Technical Docs To Rag Markdown Chunker

arezdev/technical-docs-to-rag-markdown-chunker

Convert public documentation into clean Markdown and RAG chunks with source URLs, section context and content hashes. Discover pages through sitemaps and llms.txt, preserve code and tables, and export JSON for embedding and vector database pipelines.

Generate a ready-to-upload llms.txt (and optional llms-full.txt) for any website from its sitemap and key pages, so ChatGPT, Claude and Perplexity understand and cite it. Bulk domains, no browser.