🕰️ Wayback Machine CDX Scraper

🚀 Surface every archived URL for any domain in seconds. Filter by date range, MIME type, status code, and capture digest. No API key, no registration, no manual CSV wrangling.

The Wayback Machine archives roughly 900 billion captures of the public web going back to 1996, and that backlog hides a giant catalogue of URLs, redirects, and asset paths that sites have long since broken or removed. The CDX index is the searchable layer over that backlog, and this scraper turns CDX queries into clean dataset rows. Point it at a domain like example.com, choose how broadly to match (exact URL, host, full domain), and you get back one record per unique URL with the latest snapshot timestamp, content hash, MIME type, HTTP status code at capture time, byte length, and a direct snapshot link.

This Actor is built for SEO migration audits, security recon, dead-link recovery, and academic research. It is intentionally scoped to the index alone, so it does NOT download archived page HTML, render snapshots, or mirror entire sites. That keeps runs fast and cheap, even for large prefix sweeps. Each row also carries firstSeen and lastSeen timestamps aggregated across every capture of that URL, so you can spot when a path appeared on the site and when it last responded. The output is plain JSON, ready to feed into BigQuery, a Postgres staging table, or a Make/Zapier workflow.

🎯 Target Audience and Primary Use Cases

Audience	Use Case
🔍 SEO consultants	Recover lost URLs after a site migration, find legacy paths to redirect
🛡️ Security researchers	Map a target's historical attack surface and forgotten endpoints
📰 Journalists and OSINT	Confirm a page existed before it was edited or removed
📚 Researchers and archivists	Build longitudinal datasets of a site's URL space over time
🧰 Devs and link auditors	Identify broken links that have living archived equivalents

📋 What the Wayback Machine CDX Scraper does

• 🌐 Match by URL, host, prefix, or full domain. Choose exact for one URL, prefix for a path subtree, host for everything on www.example.com, or domain for the host plus all subdomains.
• 📅 Filter by date range. Use from / to in yyyyMMdd form to constrain captures to a window (e.g. only what was archived during a redesign year).
• 🎚️ Filter by MIME type and HTTP status. Regex filters on mimetype and statuscode to keep only HTML, only images, only 2xx, only redirects, and so on. Prefix any filter with ! to invert.
• 🪞 Collapse adjacent duplicates. One capture per URL, per day, per month, or per content hash, controlled via the collapse input.
• 🔁 Auto-paginated with resume keys. Long sweeps stream through CDX resumeKey pages until maxItems unique URLs are collected.
• 🌍 Optional Apify proxy. The Internet Archive throttles datacenter IPs hard, so a residential proxy is recommended for jobs over a few thousand URLs.

Each output record represents one unique original URL. Alongside the URL, the row includes the latest capture's timestamp, MIME type, HTTP status, content hash (digest), byte length, the SURT-form urlkey, a direct web.archive.org/web/{ts}/{url} snapshot link, and aggregated firstSeen / lastSeen timestamps spanning every capture of that URL within the result set.

💡 Why it matters: the Wayback CDX index is one of the few neutral, public sources of historical URL data on the open web. Most commercial crawl tools only show you what is live today.

📊 Data fields

Each record includes: captureCount, digest, firstSeen, lastSeen, length, mimeType, originalUrl, scrapedTimestamp, statusCode, timestamp, urlkey, waybackUrl. All 12 field names come from a real production run, so what you see here is what lands in your dataset.

⚠️ Good to know: the Internet Archive rate-limits aggressively from datacenter IPs. For runs above a few thousand URLs, enable Apify Residential proxy in the input.

🚀 How to use

1. 🆔 Create a free account. Create a free account w/ $5 credit.
2. 🔎 Open the actor. Find the actor on Apify Store.
3. 📝 Fill the input form. Set urlOrDomain, optionally tighten from / to and the filters.
4. ▶️ Run. Click Start. The log streams CDX page progress and how many unique URLs have been collected.
5. ⬇️ Export. Download as JSON, CSV, Excel, or stream into a Make / Zapier workflow.

⏱️ Total time to first row: under a minute for most domains.

🔗 Recommended Actors

• 🌐 OurAirports Scraper - global airport reference data
• 🏛️ FINRA BrokerCheck Scraper - public regulatory records
• ✈️ FAA Aircraft Registry Scraper - U.S. aircraft registration index
• 📡 Yatco Scraper - yacht listings catalogue
• 🏷️ HubSpot Marketplace Scraper - app marketplace metadata

💡 Pro Tip: browse the complete ParseForge collection for more public-data scrapers built with the same conventions.

Disclaimer: This Actor is an independent project and is not affiliated with, endorsed by, or sponsored by the Internet Archive. It only reads the public CDX index. You are responsible for complying with applicable laws and the Internet Archive's terms of service when using the data downstream.

🆘 Need Help?

If you hit a bug, have questions about setup, or need a scraper we haven't built yet, open our contact form or write to parseforge@protonmail.com. We also take on paid custom data projects.

For faster answers, join our Discord. It's the best place to get support and suggest new actors.