Common Crawl Corpus Builder

Common Crawl Indexes Scraper

benthepythondev/common-crawl-indexes-scraper

Collect Common Crawl records and export id, name, to, from, cdx api, timegate as structured JSON, CSV or Excel data.

Research Corpus & Citation Graph Builder

zentrafoundry/openalex-research-graph-builder

Build research corpora and citation graph datasets from public metadata APIs.

Common Crawl Scraper

crawlerbros/common-crawl-scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Pull every web page Common Crawl captured for a domain or URL prefix. Get timestamps, MIME types, status codes, content digests, and WARC offsets to fetch original payloads. Filter by collection, MIME, and status. Export to JSON, CSV, or Excel for large-scale web research and content discovery.

Crawl public pages and extract page titles, meta descriptions, headings, readable text, source URLs, and crawl metadata.

Crawl4ai

kael_odin/crawl4ai

Extract page content (markdown/HTML/text), metadata, and link stats. Uses crawl4ai.

Fetch cross-platform prediction-market matches from Prophed Common Markets and output normalized arbitrage, liquidity, spread, and platform-comparison rows.

Torrent / BitTorrent DHT corpus search & crawl statistics

apricot_inference/fast-tracker

Search a live BitTorrent DHT metadata corpus and see how fast it is growing. Two endpoints: /search with category filters and relevance ranking, and /stats with a per-minute capture rate you can verify. No endpoint or key needed — the corpus is crawled and hosted by this Actor's operator.

Score web pages for RAG corpus quality: text length, heading structure, boilerplate hints, token estimate, metadata, and warnings.