You can access the Common Crawl Indexes Scraper programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = { "maxResults": 3 }
9
10# Run the Actor and wait for it to finish
11run = client.actor("benthepythondev/common-crawl-indexes-scraper").call(run_input=run_input)
12
13# Fetch and print Actor results from the run's dataset (if there are any)
14print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
15for item in client.dataset(run.default_dataset_id).iterate_items():
16    print(item)
17
18# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Common Crawl Indexes Scraper API in Python

The Apify API client for Python is the official library that allows you to use Common Crawl Indexes Scraper API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Common Crawl Indexes Scraper API in JavaScript

Common Crawl Indexes Scraper API through CLI

Common Crawl Indexes Scraper OpenAPI definition

Common Crawl Indexes Scraper API

Common Crawl URL Index Lookup Scraper

parseforge/common-crawl-index-scraper

Queries the Common Crawl URL index for a domain or URL and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID. Supports filters for date, status, and MIME type.

ParseForge

Common Crawl Index Catalog

zentrafoundry/common-crawl-targeted-corpus-builder

Live Common Crawl collinfo (Try is CC-MAIN-2026-34). Index catalog, not WARC text — CDX 503s from this IP. $0.01.

Zentra

Common Crawl Scraper

crawlerbros/common-crawl-scraper

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

Crawler Bros

Common Crawl Scraper (Archived HTML Finder)

igolaizola/commoncrawl-scraper

Find URLs across historical commoncrawl.org indexes and download archived HTML. Search the 10 newest or oldest snapshots, choose a crawl year, or provide up to 10 exact indexes for SEO audits, page recovery, content research, and historical web datasets.