Pay-per-result web scraper with JS rendering, CSS selector / XPath / regex extraction, schema validation, retry on failure. Use for product catalogs, competitor pricing, news aggregation, lead generation. Fast (<2s/page), respects robots.txt by default.

Universal AI Web Scraper

stanvanrooy6/universal-ai-web-scraper

Turn any website into an API. Extract structured data using plain English. Features anti-bot bypass, dynamic rendering, and web search. No coding needed.

Search the web or scrape any URL using XCrawl residential proxy network. Bypass anti-bot systems with automatic JS rendering fallback, global IP rotation, and configurable concurrency (1-20). Perfect for market research, LLM data collection, and content aggregation.

🕸️ Generic web scraper for any website with CSS selectors & JS rendering via Playwright. Static & dynamic pages. Free platform compute pricing.

Scrape GPT - Universal AI Web Scraper Agent

paradox-analytics/scrape-gpt---universal-ai-web-scraper-agent

AI-powered universal web scraper that works on ANY website without configuration. Extract data from e-commerce, news sites, social media, and more using intelligent LLM-based field mapping. Features JSON-first extraction, automatic pagination, anti-bot bypass, and cost-effective caching.

Web to Markdown API — HTML Scraper works with all sites

imapp/web-markdown-api

Convert any public URL to clean, token-efficient Markdown — purpose-built for RAG pipelines, LLM ingestion, and AI agents. Multi-layer bot evasion handles Cloudflare & JS SPAs. CSS selector filtering, metadata extraction, and token budget control included. Free trial, Pay-per-result from $0.0004.

AI Web Crawler

hounderd/ai-web-crawler

Crawl websites and extract clean, LLM-ready markdown content with stealth browser rendering, anti-bot hardening, smart content filtering, and structured metadata extraction. Built for RAG pipelines, AI agents, and data workflows.

General Web Scraper & AI Data Extraction

erng/general-web-scraper

Scrape any website and extract clean markdown/text content. Uses Playwright for JS rendering, Readability for article extraction, and outputs clean machine-readable data.

Anti-detection web scraping: fingerprint rotation, residential proxies, human-like behavior. Scrape sites that block scrapers.

Firecrawl Website Crawler

alizarin_refrigerator-owner/firecrawl-website-crawler

Enhanced Website Crawling with Superior JS Rendering Enhanced website crawler using Firecrawl's Crawl API for superior JavaScript rendering, smart rate limiting, anti-bot bypass, and clean markdown extraction.