🌐 Common Crawl Index Scraper

🚀 List every web page Common Crawl captured for a domain or URL prefix. WARC offsets included so you can fetch the original payload from S3. No API key, no registration.

The Common Crawl Index Scraper queries the public Common Crawl Index Server and returns every page Common Crawl captured for a given domain or URL prefix. Each record includes the captured URL, ISO timestamp, MIME type, HTTP status code, content digest, byte length, WARC filename, byte offset into that file, and the source collection name.

Common Crawl runs a fresh public web crawl every month and indexes the results in a sortable URL-keyed index. The dataset has powered widely-cited research, Wikipedia-grade reference work, and the training corpus for many large language models. This Actor handles collection selection, MIME and status filters, pagination, and timestamp formatting so you can focus on the data.

📋 What the Common Crawl Index Scraper does

Five filtering workflows in a single run:

• 🌐 Domain or prefix lookup. Submit a URL or prefix and pull every Common Crawl capture in the chosen collection.
• 🗂️ Collection selector. Pick a specific monthly crawl like CC-MAIN-2026-04 or default to the latest.
• 📐 Match-type control. exact, prefix, host, or domain like a CDX query.
• 📄 MIME and status filters. Restrict to HTML, JSON, image, or any specific status code.
• 📦 WARC offsets included. Every row tells you which WARC file holds the original payload and at what byte offset.

Each row reports the URL, ISO timestamp, MIME type, HTTP status, digest, byte length, WARC filename, byte offset, and the parent collection identifier.

💡 Why it matters: Common Crawl is the largest free web corpus in existence and the foundation of many open AI training datasets. Knowing whether a domain is even in the corpus, and at what depth, is a basic question for ML pretraining work, copyright analysis, and large-scale research. Direct CDX queries against the index server are doable but slow and finicky; this Actor wraps that in a clean filter UI.

📊 Data fields

Each record includes: crawlId, digest, filename, languages, length, mime, mimeDetected, offset, statusCode, timestamp, timestampIso, url, urlkey, warcUrl. These field names come straight from the actor's dataset schema, so what you see here is what lands in your dataset.

🚀 How to use

1. 🆓 Create a free Apify account. Sign up here and get $5 in free credit.
2. 🔍 Open the Actor. Search for "Common Crawl Index" in the Apify Store.
3. ⚙️ Set your inputs. Pick the URL or domain, match type, and any filters.
4. ▶️ Click Start. A 100-row run typically completes in 10 to 25 seconds.
5. 📥 Download. Export as CSV, Excel, JSON, or XML.

⏱️ Total time from sign-up to first dataset: under five minutes.

🔗 Recommended Actors

• 🕰️ Wayback Machine CDX Scraper - the Internet Archive's complementary historical web index.
• 🅱️ Bing Search Scraper - check current rank for URLs you find in CC.
• 🦆 DuckDuckGo Search Scraper - alternative SERP signal alongside crawl coverage.
• 📚 Wikipedia Pageviews Scraper - cross-reference web mentions with public-interest spikes.
• 🐙 GitHub Trending Repos Scraper - capture the developer-attention layer.

💡 Pro Tip: browse the complete ParseForge collection for more pre-built scrapers and data tools.

Common Crawl is a registered trademark of Common Crawl Foundation, a 501(c)(3) non-profit. This Actor is not affiliated with or endorsed by Common Crawl. It uses only the public Index Server endpoint and respects all published rate limits.

🆘 Need Help?

If you hit a bug, have questions about setup, or need a scraper we haven't built yet, open our contact form or write to parseforge@protonmail.com. We also take on paid custom data projects.

For faster answers, join our Discord. It's the best place to get support and suggest new actors.