Common Crawl Index Catalog

Common Crawl Indexes Scraper

benthepythondev/common-crawl-indexes-scraper

Collect Common Crawl records and export id, name, to, from, cdx api, timegate as structured JSON, CSV or Excel data.

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Supports filters for date, status, and MIME type.

Common Crawl Scraper

crawlerbros/common-crawl-scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Public Catalog Price Snapshot

zentrafoundry/sku-price-drop-alert-api

Live Shopify products.json prices (Try is Allbirds). First run is a baseline, not a drop alert. $0.01 / SKU.

Shopify products.json list vs compare-at when the catalog prints it (Try is Allbirds). Not historical MSRP proof. $0.01.

Typed rows from a public JSON catalog (Try is Allbirds): title, price number, URL. $0.002 / row.

Scrape the Internet Archive Wayback Machine CDX index: find all archived snapshots of any URL with timestamps, HTTP status codes, and MIME types.

JSON Catalog Deduplicator

zentrafoundry/dataset-deduplicator

Unique handle keys from a public JSON catalog (Try is Allbirds). Not an Apify-platform merge. $0.002 / unique row.

Wayback CDX Scraper - Archived URL and Snapshot Data

benthepythondev/wayback-cdx-scraper

Query the public Wayback CDX index for archived URLs, timestamps, MIME types, digests, response codes and replay links.