Common Crawl URL Index Lookup Scraper

Common Crawl URL Index Lookup Scraper

Look up any URL or domain in the Common Crawl index and get every crawl record, timestamp, status code, and MIME type. No API key, no rate limits, no setup. Export to CSV, JSON, Excel, or XML.

Common Crawl is a massive, open repository of web crawl data, but querying its index directly means downloading huge files and parsing them yourself. This Actor queries the Common Crawl URL index for any domain or URL and returns each matching record as a flat row, with filters for crawl date, HTTP status, and MIME type.

Who uses it	What they scrape Common Crawl for
SEO analysts	Checking when a competitor's page was last crawled and what status it returned
Researchers	Building a timeline of a website's changes across Common Crawl snapshots
Data engineers	Extracting URL lists from Common Crawl for downstream processing
Digital archivists	Finding all crawl records for a domain to reconstruct its history

What it does

This Actor looks up a URL or domain in the Common Crawl index and returns each crawl record as a flat row with timestamp, status code, MIME type, and crawl ID.

• 🔍 Flexible matching: match by exact URL, prefix, host, or domain.
• 📅 Date filters: restrict results to crawls between two dates.
• 🚦 Status and type filters: keep only records with a specific HTTP status code or MIME type.
• 🗂️ Crawl selection: query a specific crawl ID or let the Actor pick the most recent crawl automatically.
• 📦 Flat output: every record is one row, ready for CSV, JSON, Excel, or XML export.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with Common Crawl data

📈 Track competitor crawl history.

An SEO analyst enters a competitor's domain and gets every crawl record with timestamps and status codes to see how often the site is indexed.

🧪 Audit your own site's index coverage.

A webmaster looks up their domain with a status code filter to find pages that returned 404 or 500 errors in recent crawls.

📚 Build a research dataset.

A researcher queries a domain with a MIME type filter to collect all PDF documents from Common Crawl snapshots for analysis.

🗄️ Reconstruct a site's history.

A digital archivist uses prefix matching on a URL to find every crawl record for a specific section of a website over time.

Why choose this scraper

	What you get
No API key	Query the public Common Crawl index without registration or authentication
No rate limits	Run as many lookups as you need, up to one million records per run
Historical coverage	Access crawl records from multiple Common Crawl snapshots
Structured output	Get a clean, flat dataset instead of raw index files

How it compares

No other Store actor targets Common Crawl the same way, so the honest comparison is with the alternatives teams actually weigh.

	Common Crawl URL Index Lookup Scraper	Build it in-house	By hand
Setup	Run it now, zero config	Days of engineering	None, but hours per pull
When Common Crawl changes	Maintained for you	You fix it	You re-learn the page
Proxies, retries, anti-bot	Built in	Your problem	Browser only
Output	Fixed JSON schema, CSV/Excel export	Whatever you build	Copy-paste
Cost	Pay per result	Engineering time	Analyst hours

Configure the run

Drive the Actor with a URL or domain, choose a match type, and apply filters for crawl date, HTTP status code, and MIME type. Only matching records reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
  "maxItems": 10,
  "url": "apify.com",
  "matchType": "domain"
}

A larger pull:

{
  "maxItems": 200,
  "url": "apify.com",
  "matchType": "domain"
}

Pricing

Pay-per-result: $0.011 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$1.10
1,000 results	$11.00
10,000 results	$110.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the Common Crawl URL Index Lookup Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to Common Crawl through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/common-crawl-index-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check your URL or domain input and match type. If you used exact, try domain or prefix to broaden the match. Also verify your date, status code, and MIME type filters are not too restrictive.

Why are results from an old crawl?

The Actor uses the most recent crawl by default, but if you specified crawl IDs, it will only query those. Remove the crawl IDs to use the latest crawl.

Why do I get a timeout or slow response?

Querying the Common Crawl index can take time for large domains. Try narrowing your match type or adding filters to reduce the number of records.

Why is my date filter not working?

Dates must be in YYYYMMDD format, for example 20240101 for January 1, 2024. Make sure there are no dashes or slashes.

Why do I get records with different status codes?

If you do not set a status code filter, the Actor returns all records regardless of HTTP status. Add a status code filter to restrict results.

FAQ

Question	Answer
What is Common Crawl?	Common Crawl is a non-profit organization that crawls the web and publishes the resulting data as open datasets. The index lets you look up which URLs were crawled and when.
Do I need an API key or account?	No. This Actor queries the public Common Crawl index directly, so no registration or authentication is required.
How do I specify a URL or domain?	Enter a domain like apify.com, a path like example.com/blog, or a full URL like https://news.ycombinator.com. The match type controls how it is interpreted.
What does the match type mean?	exact matches the URL exactly, prefix matches URLs starting with the given string, host matches all URLs on the same host, and domain matches the domain and all subdomains.
Can I filter by crawl date?	Yes. Use the from and to date fields in YYYYMMDD format to restrict results to crawls within a specific time range.
Can I filter by HTTP status code?	Yes. Enter a status code like 200, 301, or 404 to return only records with that code.
Can I filter by MIME type?	Yes. Enter a MIME type like text/html, application/pdf, or image/jpeg to return only records of that type.
How many records can I get?	You can set the maximum number of records per run, up to one million.
Which Common Crawl crawl is used?	By default, the Actor queries the most recent crawl automatically. You can also specify one or more crawl IDs to query specific snapshots.
What output formats are supported?	The dataset can be exported to CSV, JSON, Excel, or XML.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Common Crawl Foundation. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.