Katana Web Crawler (ProjectDiscovery)

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Letterboxd API Scraper

rl1987/letterboxd-api-scraper

Scrape Letterboxd public film data: search & browse films, full film detail (genres, directors, rating, description, poster), popular reviews, list films, and optional statistics (watches/likes/ratings/fans + rating histogram). Read-only, guest access, no account.

Stealth Website Crawler

nocturne/stealth-website-crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple URLs simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

API & Endpoint Discovery

lofomachines/api-endpoint-discovery

Discover, test, and fingerprint all API endpoints from any website. Supports bulk input, OpenAPI/Swagger parsing, GraphQL introspection, and deep auth/CORS/rate-limit detection. Use as a Claude MCP tool via Apify.

FlareSolverr Standby

rl1987/flaresolverr-helper

Runs FlareSolverr (Cloudflare/DDoS-Guard bypass proxy) as an always-on Apify Standby Actor, with optional Apify Proxy.

Page Scraping Analyzer

apify/page-analyzer

Performs analysis of a webpage to figure out the best way how to scrape its data. Provide a URL and data points to find and get back a detailed dashboard showing how the data can be scraped. Works with initial and rendered HTML, JavaScript variables and dynamically loaded data.

**Claude AI Web Automation**

dtrungtin/claude-ai-web-automation

A real browser with Anthropic's Claude models to navigate any website and extract structured data — no CSS selectors or page-specific scraping code required.

AI Web Scraper with Playwright Browser (No-Code, MCP)

data_rig/ai-web-scraper

Run a real Playwright browser as an AI web scraper. Extract structured data from any site using natural language—no selectors or scripts. Handles JS-heavy pages, pagination, and interactions. Built for MCP agents like OpenCode and Claude Code.

CVE Scraper

rl1987/cve-scraper

CVE.org vulnerability info scraper