API Documentation Crawler for RAG & AI Agents

Compliance-Grade Web Intelligence for AI Agents

ai_solutionist/compliance-web-intel

The scraper AI agents trust. Extract grounded facts with citations, entities, claims & RAG chunks. Built for LangChain, LlamaIndex, AutoGPT. Quality scoring, auto-citations, 6 task modes.

NPS Park Conditions, Alerts & Data Scraper

jungle_synthesizer/nps-park-conditions-alerts-api-scraper

Scrape the NPS API for parks, alerts, campgrounds, events, news, things to do, and visitor centers. Filter by state or park code. Returns unified records with coordinates, entrance fees, hours, and more. Ideal for trip planning apps and AI agents monitoring closures and danger advisories.

Arxiv Semantic Search

draouadmohamed/arxiv-semantic-search

Scrape arXiv papers by category and find relevant research using AI-powered semantic search. Get papers from any field (AI, physics, biology, economics, etc.) with embeddings for RAG systems. Find your categories at: https://arxiv.org/category_taxonomy

OpenAPI Spec Scraper — APIs to LLM Agent Tools (MCP-Ready)

scrapersdelight/openapi-spec-scraper

Bulk-scrape machine-readable API specs from the apis.guru directory (2,500+ APIs) or any spec URL, normalized into LLM-agent-ready rows — one per endpoint: method, path, summary, parameters, auth, tags. Build agent-tool / RAG datasets in one run. $4 per 1,000 endpoints.

RAG Website Crawler - Markdown Chunks for LLMs & MCP

themineworks/rag-crawler

Crawl any website into clean, pre-chunked Markdown with per-chunk token counts for RAG pipelines, vector DBs (Pinecone, Qdrant) and LLM context. MCP-native for Claude & ChatGPT. SPA support via Playwright. Pay only for pages that crawl. A Firecrawl alternative.

YouTube Transcript API & Bulk Subtitle Downloader

tugelbay/youtube-transcript

Bulk YouTube transcript API for SRT/VTT, Markdown, JSON, and text exports with metadata for AI/RAG, research, subtitles, and content workflows. Guide: https://konabayev.com/tools/youtube-transcript-scraper/?utm_source=apify_info&utm_medium=referral&utm_campaign=youtube-transcript

Article Extraction API

tugelbay/article-extractor

Extract clean article text and metadata from URLs as Markdown, text, or HTML for RAG, AI agents, monitoring, and research. Guide: https://konabayev.com/tools/article-extractor/?utm_source=apify_info&utm_medium=referral&utm_campaign=article-extractor

Web Content Extractor API — URL to JSON

george.the.developer/web-content-extractor-api

Extract structured JSON from any webpage. Articles, products, recipes, jobs. Auto-detects content type. Returns metadata, headings, images, links. For AI agents and RAG.

GitHub Trending Scraper

automation-lab/github-trending-scraper

Scrape GitHub Trending repositories by language and time range: today, this week, or this month. Extracts repo names, star counts, forks, star gains, and top contributors. Great for dev trend tracking, tech newsletters, and investment research. Export to JSON, CSV, or Excel.

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.