Common Crawl Scraper — Archived HTML, URL Lookup & Historical Pages

Find URLs across the Common Crawl archive at commoncrawl.org and download HTML from historical web pages. Use the results for page recovery, SEO audits, website research, content comparison, and archive-based datasets.

🤖 What does Common Crawl Scraper do?

Common Crawl Scraper checks whether URLs appear in Common Crawl and returns the first matching archived snapshot from the indexes you choose to search as HTML, Markdown, plain text, or any combination.

• Find archived URLs — confirm whether a domain or page was captured
• Download historical HTML — recover page content from a matching crawl
• Choose your output — return HTML, Markdown, plain text, or multiple formats
• Choose the search period — start with the newest or oldest available indexes
• Choose a crawl year — search every available index from a specific year
• Target exact crawls — provide specific index IDs in your preferred order

Great for: SEO audits, expired-page recovery, competitor research, website history analysis, content archiving, and training-data collection.

SEO keywords: Common Crawl scraper, commoncrawl.org scraper, Common Crawl HTML extractor, download archived HTML, historical webpage scraper, Common Crawl API alternative, Apify actor.

💡 Why use Common Crawl?

• 📚 Recover missing content — retrieve archived HTML after a live page changes or disappears
• 🔎 Research website history — inspect earlier landing pages, articles, and product pages
• 📈 Support SEO analysis — investigate historical content, redirects, and page availability
• 🧠 Build research datasets — collect archived source material for analysis or machine learning
• 🗓️ Focus on a time period — search recent crawls, early crawls, or selected index IDs

🧠 Choose how indexes are searched

The Actor supports two index-selection modes:

Automatic discovery

Leave indexes empty and the Actor discovers the available Common Crawl indexes. Without year, it checks 10 indexes from the end selected by indexOrder: the 10 newest or the 10 oldest. Set year to check every available index from that year. The search stops at the first matching HTML snapshot.

Explicit indexes

Provide indexes when you already know which crawls matter. The Actor checks up to the first 10 unique IDs in the exact order supplied and ignores year and indexOrder.

🚀 How to use

1. Add the Actor – Open Common Crawl Scraper in Apify Store.
2. Add URLs – Enter full URLs or host-and-path values in urls.
3. Choose the index strategy – Keep the default 10 newest indexes, select the 10 oldest, set year for every index from one year, or provide exact indexes.
4. Run the Actor – Click Run in Apify Console or trigger it through the API.
5. Choose output formats – Keep HTML, or select Markdown, plain text, or multiple formats.
6. Download results – Export the dataset as JSON, CSV, Excel, XML, or another supported format.

💳 Pricing

Apify's Free plan includes monthly credits suitable for small tests and occasional archived-page lookups. Larger URL lists and deeper index searches use more requests, so paid plans provide more capacity for recurring research and automation.

📝 Input Parameters

Parameter	Type	Required	Description
urls	Array of strings	Yes	URLs or host-and-path values to find in Common Crawl. Empty and duplicate entries are removed.
year	Integer	No	Crawl year to search, such as 2025. Every available index from that year is checked. When omitted, 10 indexes are selected according to indexOrder. Ignored when indexes is provided.
indexOrder	String	No	Discovery order: "newest" · "oldest". Default "newest". Ignored when indexes is provided.
indexes	Array of strings	No	Exact Common Crawl index IDs, such as CC-MAIN-2026-25. Up to the first 10 unique IDs are checked in the supplied order, bypassing automatic discovery.
outputFormats	Array of strings	No	Content fields to return: "html", "markdown", and/or "text". Defaults to ["html"].

Example inputs

Recover the freshest available snapshots

{
  "urls": [
    "https://commoncrawl.org/",
    "https://example.com/about",
    "apify.com/store"
  ],
  "outputFormats": [
    "html",
    "markdown",
    "text"
  ],
  "indexOrder": "newest"
}

Search every crawl from a historical year

{
  "urls": [
    "https://example.com/",
    "https://example.com/products"
  ],
  "year": 2024,
  "indexOrder": "oldest"
}

Search selected crawls in a custom order

{
  "urls": [
    "https://example.com/blog/launch-announcement"
  ],
  "indexes": [
    "CC-MAIN-2024-10",
    "CC-MAIN-2025-30",
    "CC-MAIN-2026-25"
  ]
}

📊 Output / Results

Each requested URL produces one dataset item:

• requestedUrl (string) — the URL or host-and-path value submitted
• found (boolean) — whether a matching archived HTML page was found
• index (string, optional) — the Common Crawl index containing the match
• url (string, optional) — the resolved URL stored in the crawl
• html (string, optional) — the archived HTML content when selected in outputFormats
• markdown (string, optional) — Markdown converted from the archived HTML when selected in outputFormats
• text (string, optional) — plain text converted from the archived HTML when selected in outputFormats
• error (string, optional) — details when a lookup cannot be completed

{
  "requestedUrl": "https://commoncrawl.org/",
  "found": true,
  "index": "CC-MAIN-2026-25",
  "url": "https://commoncrawl.org/",
  "html": "<!DOCTYPE html><html lang=\"en-US\"><head><title>Common Crawl</title>...</head><body>...</body></html>"
}

When no snapshot is found in the selected indexes:

{
  "requestedUrl": "https://example.com/page-that-was-never-crawled",
  "found": false
}

🧭 Tips / Common Recipes

• Recover recent content: omit year and choose indexOrder: "newest" to check the 10 newest indexes.
• Find the earliest available page: choose indexOrder: "oldest"; the first match is the earliest matching crawl index.
• Search one historical year: set year: 2024 to check every available index from 2024.
• Compare selected periods: run the same URL with different one-item indexes lists and compare the returned HTML.
• Keep explicit lists focused: only the first 10 unique indexes are checked.
• Reduce unnecessary lookups: use exact page URLs and omit year when you only care about recent history.
• Search without a scheme: values such as example.com/about are accepted alongside full URLs.
• Reduce result size: omit "html" from outputFormats when you only need Markdown or plain text.

⚙️ Best Practices

• Use canonical URLs when possible to reduce ambiguous or duplicate matches.
• Choose indexes deliberately because the Actor returns the first match, not every historical capture.
• Plan for large results because returning multiple formats, especially full HTML, can make dataset exports significantly larger.
• Retry individual failures when an item contains error, as archive availability and network conditions can vary.
• Remember that snapshots are historical and may not represent the current live page.

🌍 Proxy Usage

Common Crawl is a public archive, so most runs do not require special proxy routing. If your network cannot reach the archive reliably, Apify proxy settings can improve connectivity; residential targeting is usually unnecessary.

⚖️ Legal & Ethical Considerations

• Respect applicable website terms, archive policies, and downstream content restrictions.
• Protect personal information and comply with privacy requirements such as GDPR or CCPA.
• Use reasonable workloads and avoid abusive or unnecessarily repetitive archive requests.
• Preserve attribution and ownership when publishing, redistributing, or analyzing archived content.
• Review usage rights before using archived pages in commercial datasets or machine-learning projects.

This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Common Crawl. All trademarks belong to their respective owners.

❓ FAQ

Does the Actor fetch the live page? No. It returns HTML stored in Common Crawl, not the current version of the website.

Which index is searched by default? The Actor checks up to the 10 newest indexes and stops at the first matching HTML snapshot.

What happens when I set year? The Actor checks every available index from that year in the selected indexOrder and stops at the first matching HTML snapshot.

What happens when I provide indexes? Automatic discovery is skipped. Up to the first 10 unique IDs are checked in order, while year and indexOrder are ignored.

Does the Actor return every snapshot of a URL? No. It returns the first matching HTML snapshot found using your chosen index order.

Why does a result contain error instead of HTML? The archive lookup or download could not be completed for that URL. Retry the item, verify any explicit index IDs, and check your proxy settings if failures continue.

🛟 Support

Need a different archive field, a bulk-domain workflow, or multiple snapshots per URL? Open an issue or contact me at https://igolaizola.com/#contact