Common Crawl Index Scraper

What does Common Crawl Index Scraper do?

Scrape Common Crawl index metadata. Extract domain coverage, crawl dates, page counts, and segment info. It runs on the Apify platform and delivers structured data in JSON, CSV, or Excel format, ready for analysis, integration, or automation workflows. Common Crawl Index Scraper handles pagination, retries, and proxy rotation automatically so you can focus on using the data.

Why use Common Crawl Index Scraper?

• No coding required — configure inputs in a simple web UI and click Start
• Export anywhere — download results as JSON, CSV, or Excel, or connect via API
• Scheduled runs — set up recurring scrapes to keep your data fresh (hourly, daily, weekly)
• Scalable — process hundreds or thousands of items with automatic proxy rotation and retry logic
• Integrations — connect to Google Sheets, Slack, Zapier, Make, webhooks, and more through the Apify platform

How to use Common Crawl Index Scraper

1. Navigate to the Common Crawl Index Scraper page on Apify Store and click Try for free
2. Configure your input parameters (see Input Configuration below)
3. Click Start and wait for the run to complete
4. View results in the Output tab — use the formatted table or switch to raw JSON
5. Download your data as JSON, CSV, or Excel, or access it via the Apify API

Input configuration

Field	Type	Description	Default
URLs	array	Common Crawl index pages to scrape	['https://index.commoncrawl.org/collinfo.json']
Max Results	integer	Maximum number of results	100

Output data

The actor stores results in a dataset. Each item in the dataset represents one extracted record with structured fields. You can preview the data in the Output tab's formatted table view.

Key output fields include: Name, Timegate, Scraped At, ID, Cdx API.

Example output:

{
  "name": "Example Name",
  "timegate": "2026-01-15T10:30:00Z",
  "scrapedAt": "Example Scraped At",
  "id": "Example ID",
  "cdxApi": "Example Cdx API"
}

Each run also produces an execution log with detailed information about pages processed, items extracted, and any errors encountered.

Cost of usage

Common Crawl Index Scraper uses Pay-Per-Event pricing (Utility tier). Each successfully extracted result costs approximately $0.0003 ($0.30 per 1,000 results).

On a free Apify plan ($5/month platform credit), you can extract approximately 16,666 results per month.

Example: Extracting 1,000 results would cost approximately $0.30.

Tips and advanced usage

• Proxy configuration: This actor uses lightweight HTTP requests for fast, efficient scraping. For sites with rate limiting, the actor automatically rotates proxies.
• Large datasets: For runs with thousands of results, increase the memory allocation in Run Options to speed up processing. The actor automatically manages request queues and pagination.
• Scheduled runs: Use Apify Schedules to run this actor on a recurring basis. Combined with integrations (webhooks, Google Sheets, Slack), you can build automated data pipelines that keep your datasets up to date.
• API access: Every dataset is accessible via the Apify API. Use the REST API or official Python/JavaScript clients to integrate results directly into your applications.

Useful Links

• Common Crawl
• Apify Proxy

Related Actors:

• Wikipedia Category Extractor

• More actors by consummate_mandala

• GitHub