🕰️ Wayback Machine CDX Scraper

🚀 Surface every archived URL for any domain in seconds. Filter by date range, MIME type, status code, and capture digest. No API key, no registration, no manual CSV wrangling.

🕒 Last updated: 2026-07-14 · 📊 12 fields per record · Powered by the Internet Archive CDX index · No browser, pure HTTP · Up to 1M URLs per run

The Wayback Machine archives roughly 900 billion captures of the public web going back to 1996, and that backlog hides a giant catalogue of URLs, redirects, and asset paths that sites have long since broken or removed. The CDX index is the searchable layer over that backlog, and this scraper turns CDX queries into clean dataset rows. Point it at a domain like example.com, choose how broadly to match (exact URL, host, full domain), and you get back one record per unique URL with the latest snapshot timestamp, content hash, MIME type, HTTP status code at capture time, byte length, and a direct snapshot link.

This Actor is built for SEO migration audits, security recon, dead-link recovery, and academic research. It is intentionally scoped to the index alone, so it does NOT download archived page HTML, render snapshots, or mirror entire sites. That keeps runs fast and cheap, even for large prefix sweeps. Each row also carries firstSeen and lastSeen timestamps aggregated across every capture of that URL, so you can spot when a path appeared on the site and when it last responded. The output is plain JSON, ready to feed into BigQuery, a Postgres staging table, or a Make/Zapier workflow.

🎯 Target Audience and Primary Use Cases

Audience	Use Case
🔍 SEO consultants	Recover lost URLs after a site migration, find legacy paths to redirect
🛡️ Security researchers	Map a target's historical attack surface and forgotten endpoints
📰 Journalists and OSINT	Confirm a page existed before it was edited or removed
📚 Researchers and archivists	Build longitudinal datasets of a site's URL space over time
🧰 Devs and link auditors	Identify broken links that have living archived equivalents

---

📋 What the Wayback Machine CDX Scraper does

• 🌐 Match by URL, host, prefix, or full domain. Choose exact for one URL, prefix for a path subtree, host for everything on www.example.com, or domain for the host plus all subdomains.
• 📅 Filter by date range. Use from / to in yyyyMMdd form to constrain captures to a window (e.g. only what was archived during a redesign year).
• 🎚️ Filter by MIME type and HTTP status. Regex filters on mimetype and statuscode to keep only HTML, only images, only 2xx, only redirects, and so on. Prefix any filter with ! to invert.
• 🪞 Collapse adjacent duplicates. One capture per URL, per day, per month, or per content hash, controlled via the collapse input.
• 🔁 Auto-paginated with resume keys. Long sweeps stream through CDX resumeKey pages until maxItems unique URLs are collected.
• 🌍 Optional Apify proxy. The Internet Archive throttles datacenter IPs hard, so a residential proxy is recommended for jobs over a few thousand URLs.

Each output record represents one unique original URL. Alongside the URL, the row includes the latest capture's timestamp, MIME type, HTTP status, content hash (digest), byte length, the SURT-form urlkey, a direct web.archive.org/web/{ts}/{url} snapshot link, and aggregated firstSeen / lastSeen timestamps spanning every capture of that URL within the result set.

💡 Why it matters: the Wayback CDX index is one of the few neutral, public sources of historical URL data on the open web. Most commercial crawl tools only show you what is live today.

---

🎬 Full Demo

🚧 Coming soon: a 3-minute walkthrough showing a sample SEO recovery sweep end to end.

---

⚙️ Input

Field	Type	Required	Description
urlOrDomain	string	yes	Target URL, host, or domain. Examples: example.com, https://example.com/blog/, sub.example.com
matchType	enum	no	One of exact, prefix, host, domain. Defaults to domain.
maxItems	integer	no	Max unique URLs to return. Free tier capped at 10. Paid up to 1,000,000.
from	string	no	Inclusive lower bound on capture timestamp. Format yyyy[MM[dd[HH[mm[ss]]]]].
to	string	no	Inclusive upper bound on capture timestamp. Same format as from.
mimeFilter	string	no	Regex on the capture's MIME type. Prefix with ! to invert.
statusFilter	string	no	Regex on the capture's HTTP status code. Prefix with ! to invert.
collapse	string	no	CDX collapse field. urlkey, digest, timestamp:8 (one per day), etc.
proxyConfiguration	object	no	Apify proxy configuration. Residential recommended for large sweeps.

Example: a 2020 to 2024 sweep of all archived HTML pages on apify.com and its subdomains, with one URL per day:

{
    "urlOrDomain": "apify.com",
    "matchType": "domain",
    "from": "2020",
    "to": "2024",
    "mimeFilter": "^text/html",
    "statusFilter": "^2",
    "collapse": "timestamp:8",
    "maxItems": 500
}

Example: every PDF ever archived under irs.gov/pub/:

{
    "urlOrDomain": "irs.gov/pub/",
    "matchType": "prefix",
    "mimeFilter": "application/pdf",
    "maxItems": 1000
}

⚠️ Good to know: the Internet Archive rate-limits aggressively from datacenter IPs. For runs above a few thousand URLs, enable Apify Residential proxy in the input.

---

📊 Output

Each dataset row is one unique original URL. Aggregations like firstSeen, lastSeen, and captureCount are computed across all CDX rows seen during the run.

🧾 Schema

Field	Type	Example
🌐 originalUrl	string	https://www.apify.com/about
🕒 timestamp	string (ISO)	2024-09-12T08:31:44Z
🔗 waybackUrl	string (URL)	https://web.archive.org/web/20240912083144/https://www.apify.com/about
📄 mimeType	string	text/html
🔢 statusCode	integer	200
🧬 digest	string	7E5K2ZFDPQHM6XTMG5BNYE3LK4QFZJC2
📦 length	integer	34921
🌅 firstSeen	string (ISO)	2018-03-01T11:08:22Z
🌇 lastSeen	string (ISO)	2024-09-12T08:31:44Z
📊 captureCount	integer	41
🔑 urlkey	string (SURT)	com,apify)/about
⏱️ scrapedTimestamp	string (ISO)	2026-04-30T20:11:08Z

📦 Sample records

---

✨ Why choose this Actor

	Capability
🪶	No browser overhead. Pure HTTP against the public CDX endpoint. Cheap to run, fast to finish.
🌐	Four match modes. From a single URL up to a whole domain plus subdomains, no separate runs.
🎛️	Full filter set. Date range, MIME regex, status regex, and collapse, all server-side.
📜	Resume-key pagination. Streams long result sets without holding everything in memory at once.
🌅	First-seen and last-seen. Aggregates all captures of each URL, so you see how long a path lived.
🌍	Apify proxy aware. Residential or datacenter rotation for sweeps that get rate-limited.
💾	Clean dataset shape. Twelve well-typed fields, no nulls on present data, plus a direct snapshot link.

📊 The CDX index covers roughly 900 billion captures and is one of the largest public archives of historical URLs available.

---

📈 How it compares to alternatives

Approach	Cost	Coverage	Refresh	Filters	Setup
⭐ Wayback Machine CDX Scraper (this Actor)	Apify usage only	Whole CDX, 1996-present	Live	Date, MIME, status, collapse	None, run from console
Manual curl to CDX	Free	Same	Live	Same	DIY pagination, retries, parsing
Paid live web indices	High monthly	Live web only, no archive depth	Live	Per-vendor	Account, billing, API key
Self-hosted CDX mirror	Free, time-heavy	Whole CDX	Stale	Same	Hours of setup, big disk

For most teams the calculus is simple: a hosted scraper that returns clean JSON is worth more than the time spent re-implementing CDX pagination and rate-limit handling.

---

🚀 How to use

1. 🆔 Create a free account. Create a free account w/ $5 credit.
2. 🔎 Open the actor. Find the actor on Apify Store.
3. 📝 Fill the input form. Set urlOrDomain, optionally tighten from / to and the filters.
4. ▶️ Run. Click Start. The log streams CDX page progress and how many unique URLs have been collected.
5. ⬇️ Export. Download as JSON, CSV, Excel, or stream into a Make / Zapier workflow.

⏱️ Total time to first row: under a minute for most domains.

---

💼 Business use cases

🔎 SEO and migration audits Recover URLs lost during a site rebuild or platform move Build a redirect map from archived 200 responses Prove what a page used to claim before legal forced an edit Compare current sitemap coverage against historical crawl footprint	🛡️ Security recon Enumerate forgotten endpoints, staging hostnames, deprecated APIs Map a target's historical surface during red-team scoping Find leaked PDFs, configs, and backups in archived asset paths Track when sensitive paths first appeared and when they vanished
📰 Journalism and OSINT Confirm a page or claim existed before it was edited or scrubbed Build evidence packs with cited snapshot URLs Trace removed press releases, deleted bios, and archived comments Time-stamp narrative changes across a campaign or controversy	📚 Research and archiving Quantify how a domain's URL space evolved year over year Seed academic crawls with the full historic URL set Rebuild dead reading lists by mapping each link to a live snapshot Generate longitudinal datasets for digital humanities work

---

🌟 Beyond business use cases

Data like this powers more than commercial workflows. The same structured records support research, education, civic projects, and personal initiatives.

🎓 Research and academia Empirical datasets for papers, thesis work, and coursework Longitudinal studies tracking changes across snapshots Reproducible research with cited, versioned data pulls Classroom exercises on data analysis and ethical scraping	🎨 Personal and creative Side projects, portfolio demos, and indie app launches Data visualizations, dashboards, and infographics Content research for bloggers, YouTubers, and podcasters Hobbyist collections and personal trackers
🤝 Non-profit and civic Transparency reporting and accountability projects Advocacy campaigns backed by public-interest data Community-run databases for local issues Investigative journalism on public records	🧪 Experimentation Prototype AI and machine-learning pipelines with real data Validate product-market hypotheses before engineering spend Train small domain-specific models on niche corpora Test dashboard concepts with live input

---

🔌 Automating Wayback Machine CDX Scraper

Run this Actor on a schedule, from your own backend, or as part of a larger pipeline.

• Node.js via the Apify JS client
• Python via the Apify Python client
• Docs: Apify Actor API reference

Schedules are first-class on Apify. Set a cron, point it at this Actor's input, and your dataset stays fresh without any glue code.

---

❓ Frequently Asked Questions

---

🔌 Integrate with any app

• Make - drop the Actor into a no-code automation
• Zapier - trigger Zaps from each finished run
• n8n - self-hostable workflow automation
• Slack - send completion notifications and dataset links to a channel
• Webhooks - POST run events to any endpoint
• Google Sheets - mirror the dataset into a Sheet for collaborators

---

🔗 Recommended Actors

• 🌐 OurAirports Scraper - global airport reference data
• 🏛️ FINRA BrokerCheck Scraper - public regulatory records
• ✈️ FAA Aircraft Registry Scraper - U.S. aircraft registration index
• 📡 Yatco Scraper - yacht listings catalogue
• 🏷️ HubSpot Marketplace Scraper - app marketplace metadata

💡 Pro Tip: browse the complete ParseForge collection for more public-data scrapers built with the same conventions.

---

🆘 Need Help? Open our contact form

---

Disclaimer: This Actor is an independent project and is not affiliated with, endorsed by, or sponsored by the Internet Archive. It only reads the public CDX index. You are responsible for complying with applicable laws and the Internet Archive's terms of service when using the data downstream.