​AI-Ready Documentation Crawler

Llm Ready Documentation Scraper

direct_duty/llm-ready-documentation-scraper

Developers and AI agents need to read documentation (e.g. Stripe Docs, Next.js Docs), but standard scrapers return noisy HTML that includes: navigation bars headers / footers ads / cookie banners This Actor must return pure content-only Markdown, suitable for vectorization and semantic search.

Extract clean, RAG-optimized Markdown from any technical documentation. Built for LLMs and AI agents. No noise, just high-fidelity data.

Context Layer

evertools/context-layer

Transforms documentation sites into a clean, structured context layer for AI systems—handling crawling, extraction, intelligent chunking, and optional enrichment for RAG, fine-tuning, and semantic search.

Markdownify MCP Server

crawlerbros/markdownify-mcp-server

Convert any webpage to clean, formatted Markdown perfect for AI consumption. Ideal for building knowledge bases, documentation scrapers, and content migration tools.

AI Content Scraper & Cleaner — Scrapes structured content (documentation, articles, FAQs, blog posts) and converts it into clean, normalized JSON datasets for LLM training. Extracts text, detects content types, estimates tokens, and removes boilerplate to produce ready-to-use training data.

Loom Scraper

automation-lab/loom-scraper

Extract Loom video transcripts, metadata, and direct download URLs. All-in-one: transcript with timestamps + owner/view data + MP4 URL. Pure HTTP — fast and cheap. No API key needed.

/llms.txt Generator

jakub.kopecky/llmstxt-generator

The /llms.txt Generator 🕸️📄 extracts website content to create an llms.txt file for AI apps 🤖✨ like LLM fine-tuning and indexing. Output is available 📥 in the Key-Value Store for easy download and integration into workflows. 🚀

Website Content Crawler Pro

datascoutapi/website-content-crawler-pro

Crawl websites and extract clean, structured content in Markdown, JSON, or plain text for AI models, LLMs, vector DBs, or RAG pipelines. Fast, reliable, and stealthy, with bulk processing, advanced metadata extraction, and seamless integration with LangChain, LlamaIndex, and AI workflows.

Reddit Scraper

automation-lab/reddit-scraper

Extract Reddit posts, comments, and user profiles from any subreddit. No login, no API key, no code needed. Filter by keywords, sort by new/hot/top. AI-ready JSON output for LLM pipelines. Export to CSV, Excel, or connect via API.

Instagram Hashtag Scraper

scrapesmith/instagram-hashtag-scraper

⚡ Instagram Hashtag Scraper – Extract posts & reels from any public hashtag. Get captions, likes, views, comments, timestamps & media URLs. Bulk hashtag support, no login needed. Export to JSON, CSV or Excel. $0.50/1,000 results.