Scrape any site, Anti-Bot Proxy, JS Render & AI

General Web Scraper & AI Data Extraction

erng/general-web-scraper

Scrape any website and extract clean markdown/text content. Uses Playwright for JS rendering, Readability for article extraction, and outputs clean machine-readable data.

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

Best Web Scraper API

crawlkit/crawlkit-scrape-api

Scrape any website and get clean markdown, HTML, metadata and links. Powered by CrawlKit.sh - supports stealth mode for anti-bot protection.

Web to Markdown API — HTML Scraper works with all sites

imapp/web-markdown-api

Convert any public URL to clean, token-efficient Markdown — purpose-built for RAG pipelines, LLM ingestion, and AI agents. Multi-layer bot evasion handles Cloudflare & JS SPAs. CSS selector filtering, metadata extraction, and token budget control included. Free trial, Pay-per-result from $0.0004.

Firecrawl Website Crawler

alizarin_refrigerator-owner/firecrawl-website-crawler

Enhanced Website Crawling with Superior JS Rendering Enhanced website crawler using Firecrawl's Crawl API for superior JavaScript rendering, smart rate limiting, anti-bot bypass, and clean markdown extraction.

A pay-per-page web scraper that delegates to Apify Web Scraper engine. Bring your own URL and JavaScript pageFunction. Anti-block, proxy rotation, JS rendering included. Only pay for successful pages.

AI agents that need web data without anti-bot headaches. 20 tools for API-based web scraping, crawl, search, and extract — no proxy rotation, no stealth needed.

Web Scraper API - Anti-Bot Profile Selection, BYO Proxy

fenlo-services/stealth-scraper

Best-effort page fetcher that runs a quick anti-bot recon, auto-selects the matching engine request profile, and fetches the page through your own proxy. Returns HTML length, detected stack, and latency.

Anti-Bot Detection API - Batch Website Bot-Protection Profiler

fenlo-services/recon-meta

Detect a site's anti-bot stack (Cloudflare, DataDome, Akamai, PerimeterX, Imperva) and return a structured, batch-friendly pre-flight defense fingerprint as JSON.

Website Content Crawler Pro — Markdown for AI & RAG

hipersoft/web-content-crawler-pro

Crawl any website into clean Markdown or text for LLMs, RAG and AI. HTTP + JS-browser modes, main-content (readability) extraction, sitemaps, URL glob filters, cookie-banner removal and CSS selectors. No login, no API key.