Common Crawl Indexes Scraper

benthepythondev/common-crawl-indexes-scraper

Collect Common Crawl records and export id, name, to, from, cdx api, timegate as structured JSON, CSV or Excel data.

Research Corpus & Citation Graph Builder

zentrafoundry/openalex-research-graph-builder

Build research corpora and citation graph datasets from public metadata APIs.

Common Crawl Scraper

crawlerbros/common-crawl-scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

Crawl public pages and extract page titles, meta descriptions, headings, readable text, source URLs, and crawl metadata.

Crawl4ai

kael_odin/crawl4ai

Extract page content (markdown/HTML/text), metadata, and link stats. Uses crawl4ai.

Fetch cross-platform prediction-market matches from Prophed Common Markets and output normalized arbitrage, liquidity, spread, and platform-comparison rows.

Score web pages for RAG corpus quality: text length, heading structure, boilerplate hints, token estimate, metadata, and warnings.

YATCO Builder Search Scraper - Cheap 🏗️🔍

scrapestorm/yatco-builder-search-scraper---cheap

🔍 Scrape Mass / Bulk YATCO Yacht Builders Enter your builder search URL to collect yacht builder listings at scale from YATCO including builder name, location, year established, construction type & builder profile URL 🏗️🛠️ Perfect for marine industry research & yachting supply chain analysis 📊