🕰️ Wayback Machine URL Extractor — Archived & Historical URLs via the CDX API

Recover every historical URL a website has ever published — straight from the Internet Archive's Wayback Machine. This Wayback Machine URL extractor queries the public CDX API to pull the full archived URL inventory for any domain — including pages that were deleted, renamed, or lost in a migration. Feed in one domain and get back up to tens of thousands of unique URLs, each with its capture date, archived HTTP status, MIME type, content digest, and a direct Wayback snapshot link.

Point it at a single domain and it pulls the entire historical URL footprint automatically — no writing raw CDX queries by hand, no pagination logic, no rate-limit headaches. Whether you're rebuilding a redirect map after a site move, recovering deleted content, doing OSINT on a target's history, or reclaiming lost backlinks, this is the Internet Archive URL extractor that does it at scale, one clean row per archived URL, with no API key and no login.

🏆 Why this Wayback Machine scraper?

full historical URL inventory back to 1996 · official Internet Archive CDX API · date-range & HTTP-status filters · subdomain / host / prefix matching · direct web.archive.org snapshot links · streamed pagination for 100k+ URL domains · no API key / no login · export to JSON / CSV / Excel

---

✨ What this Actor does

• 🕰️ Full historical URL inventory — pulls every unique URL the Wayback Machine has on record for a domain, going back to 1996.
• 🔑 No API key required — uses the open Internet Archive CDX API; no auth, no token, no login.
• 🌐 Subdomain, host & prefix matching — capture a host plus all subdomains and paths, narrow to a single exact host, or match a specific path prefix via matchType.
• 📅 Date-range filtering — restrict to snapshots captured between two dates with fromDate / toDate (YYYYMMDD).
• ✅ Status-code filtering — keep only 200 OK captures and drop dead or redirected ones with filterStatus.
• 🔗 Direct snapshot links — every row includes a ready-to-open web.archive.org/web/... URL for the exact archived capture.
• 🌊 Streamed pagination — pages through massive result sets with the CDX resumeKey mechanism, so memory stays flat even on 100k+ URL domains.
• 🔢 Result caps — set maxResults per domain, or 0 for unlimited.
• 📋 Multiple domains per run — process a whole list of domains in one go.
• 📤 Export-ready — JSON, CSV, and Excel output via the Apify Dataset or REST API.

🚀 Quick start (3 steps)

1. Configure — add one or more domains to Domains (e.g. nasa.gov, bbc.com). Optionally pick a matchType, set a date range, filter by status code, or raise maxResults.
2. Run — click Start. The Actor builds the CDX query, pages through the archive, and streams archived URLs into the dataset.
3. Get your data — export the archived URL list as JSON, CSV, or Excel from the Output / Dataset tab, or open any row's snapshotUrl to view the archived page.

📥 Input

Give the Actor at least one domain; every other field is an optional filter. Here are three ready-to-run scenarios:

Scenario A — Full historical inventory of a domain

{
  "domains": ["nasa.gov"],
  "matchType": "subdomains",
  "maxResults": 0,
  "proxyConfiguration": { "useApifyProxy": true }
}

Scenario B — Redirect map: only live (200) pages from a date window

{
  "domains": ["example.com"],
  "matchType": "subdomains",
  "fromDate": "20100101",
  "toDate": "20201231",
  "filterStatus": "200",
  "maxResults": 5000
}

Scenario C — Several domains, exact host only

{
  "domains": ["bbc.com", "cnn.com"],
  "matchType": "host",
  "maxResults": 10000
}

Field	Type	Description
domains	array	Required. One or more domains or full URLs (e.g. nasa.gov, bbc.com, https://example.com). Scheme, www., and paths are normalized automatically — no trailing wildcards.
matchType	enum	How the domain is matched: subdomains (host + all subdomains + paths, broadest — default), host (exact host only), domain (exact domain), prefix (URLs starting with a path prefix).
fromDate	string	Optional YYYYMMDD lower bound on capture date (e.g. 20100101). Empty = no lower bound.
toDate	string	Optional YYYYMMDD upper bound on capture date (e.g. 20201231). Empty = no upper bound.
filterStatus	string	Optional — only return captures with this HTTP status (e.g. 200 to exclude dead/redirected captures).
maxResults	integer	Max unique URLs per domain. 0 = unlimited (large sites can yield hundreds of thousands). Default 5000.
proxyConfiguration	object	Apify Proxy settings, recommended to avoid rate limiting from the Internet Archive. Defaults to Apify Proxy.

📤 Output

Every run produces an Archived URLs dataset — one row per unique archived URL, deduplicated so each URL appears once rather than once per capture. Here is a trimmed, realistic sample (these are exactly the fields the Actor emits):

[
  {
    "domain": "nasa.gov",
    "url": "http://www.nasa.gov/mission_pages/station/main/index.html",
    "timestamp": "20120114043915",
    "capturedAt": "2012-01-14T04:39:15.000Z",
    "statusCode": "200",
    "mimeType": "text/html",
    "digest": "AB23CD45EF67GH89IJ01KL23MN45OP67",
    "snapshotUrl": "https://web.archive.org/web/20120114043915/http://www.nasa.gov/mission_pages/station/main/index.html"
  },
  {
    "domain": "nasa.gov",
    "url": "http://www.nasa.gov/topics/earth/features/index.html",
    "timestamp": "20150803121044",
    "capturedAt": "2015-08-03T12:10:44.000Z",
    "statusCode": "301",
    "mimeType": "text/html",
    "digest": "ZZ98YY76XX54WW32VV10UU98TT76SS54",
    "snapshotUrl": "https://web.archive.org/web/20150803121044/http://www.nasa.gov/topics/earth/features/index.html"
  }
]

🔍 How it works

1. Each domain you provide is normalized — scheme, www., paths, and wildcards are reduced to a bare host.
2. A CDX API query is built from your matchType, date range, and status filter, requesting the original, timestamp, statuscode, mimetype, and digest fields with collapse=urlkey so each URL appears once instead of returning every capture of it.
3. Results are paged with the CDX showResumeKey / resumeKey mechanism, and each page is pushed to the dataset in a batch — so even domains with hundreds of thousands of archived URLs stream out without exhausting memory.
4. For every row, a direct snapshotUrl is constructed in the https://web.archive.org/web/<timestamp>/<original-url> form so you can open the exact archived page.
5. Slow responses, 5xx, and 429 errors are retried with exponential backoff on a fresh proxy IP — the CDX index can be slow, so retries keep large runs reliable.

💡 Use cases

• 🔁 SEO migration & redirect maps — recover lost/old URLs after a site move and rebuild a complete 301 redirect map so you don't lose link equity.
• 📄 Content recovery — find and restore blog posts, product pages, or docs that were deleted but still live in the archive.
• 🕵️ OSINT & research — enumerate a target domain's historical footprint: old endpoints, removed pages, and forgotten subdomains.
• 🔗 Link reclamation — surface old URLs that still earn backlinks, then redirect them to reclaim the link value.
• 🗂️ Finding old endpoints — reveal admin paths, legacy APIs, and orphaned pages that no longer appear on the live site.
• 🏛️ Web archaeology & competitive research — reconstruct how a competitor's URL structure and content changed across years of snapshots, and build URL/MIME/capture-history datasets for analysis.

👥 Who uses it

SEO specialists & consultants · web-migration engineers · security researchers & OSINT analysts · penetration testers · digital archivists · content & growth teams · journalists · data scientists building historical web datasets

💰 Pricing

This Actor uses Apify's pay-per-result model — you pay for the archived URLs it returns plus the underlying platform compute, with no monthly subscription. The Internet Archive CDX API itself is free and needs no key; you only pay for the Apify run. Use maxResults, date ranges, and status filters to keep large-domain runs lean. See the Pricing tab on the Actor's Apify page for exact, up-to-date figures.

❓ Frequently Asked Questions

Is this a Wayback Machine / CDX API alternative?

It's a ready-made front end for the Internet Archive CDX API. Instead of hand-crafting CDX queries, handling resumeKey pagination, and de-duplicating captures yourself, you send a domain and get a clean, structured, export-ready dataset of archived URLs.

Can I use it without an API key or login?

Yes. The Internet Archive CDX API is public and requires no API key and no login. You only pay for the Apify platform usage of the run itself.

How do I get all URLs of a website from the Wayback Machine?

Add the domain to Domains, leave matchType on subdomains, set maxResults to 0 for everything, and run it. The Actor queries the CDX API and returns one row per unique archived URL.

Can I find old or deleted pages of a domain?

Yes — that's the core use case. The Wayback Machine keeps URLs even after they're removed from the live site, so deleted blog posts, retired product pages, and old endpoints all appear in the results with a snapshotUrl to view them.

How do I export archived URLs to CSV or JSON?

Run the Actor, then download the dataset as CSV, JSON, or Excel (or pull it via the REST API). Every archived URL is one row, so it drops straight into a spreadsheet or pipeline.

Is it legal to extract archived URLs?

The Actor only reads publicly available archive data from the Internet Archive — the same snapshots anyone can browse on web.archive.org. It performs no login and accesses no private data. You remain responsible for using the recovered URLs in line with applicable law and the source's terms.

How much data can it return?

Up to tens of thousands per domain — set maxResults to 0 for unlimited. Results stream to the dataset in pages via the CDX resumeKey, so even 100k+ URL domains run without memory issues.

Can I filter by date or HTTP status?

Yes — set fromDate / toDate (YYYYMMDD) to restrict to a capture window, and filterStatus (e.g. 200) to keep only captures with a specific HTTP status.

How do I build a 301 redirect map after a site migration?

Extract all archived URLs for the domain with filterStatus set to 200, then map each recovered old URL to its new destination to rebuild link equity.

Why are some statusCode values -?

The Wayback index sometimes records captures without a stored status code (e.g. revisit records). Those rows are still valid archived URLs.

🔗 More website & lead-gen tools by logiover

Actor	What it does
Sitemap to URL Crawler	Extract all URLs from any website's sitemap.xml
Website Link Graph Crawler	Crawl internal links and map a site's full link graph
Subdomain Finder	Enumerate a domain's subdomains
Website SEO Audit Crawler	Full on-page SEO audit across an entire site
URL to Markdown	Convert any page to clean, RAG-ready Markdown
Bulk URL Status Checker	Check HTTP status codes for a list of URLs in bulk
Broken Link Checker	Crawl a site and find dead links with HTTP status codes
Certificate Transparency Monitor	Discover hosts/subdomains from CT logs
Bulk WHOIS & RDAP Lookup	Bulk domain ownership & registration data
Website Contact Scraper	Extract emails, phones, and socials from websites
Website Tech Stack Detector	Detect the technologies a site is built with
Website Change Monitor	Detect and track changes to any web page

👉 Browse all logiover scrapers on Apify Store — 180+ actors across real estate, jobs, crypto, social media & B2B data.

⏰ Scheduling & integration

• Schedule — use Apify Schedules to re-run the extractor periodically and track how a domain's archived footprint grows over time.
• Export — download any run as JSON, CSV, or Excel, or query the dataset via the REST API.
• Automate — pipe results into Make, n8n, or Zapier to feed recovered URLs into a redirect-map builder, status checker, or Google Sheet.
• API & webhooks — start runs and pull results with the Apify API, and attach webhooks so downstream steps fire automatically when a run finishes.

⭐ Support & feedback

Hit an issue or want a new filter exposed? Open a ticket on the Actor's Issues tab and include the domain, the matchType you used, and the volume you expected. If this Actor saves you time, a ★★★★★ review on its Apify Store page is hugely appreciated and helps others find it.

⚖️ Legal

This Actor reads only publicly available archive data served by the Internet Archive's public CDX API — the same snapshots any visitor can browse on web.archive.org. It performs no login and accesses no private data. You are responsible for using the recovered URLs and snapshots in compliance with applicable laws, the Internet Archive's terms, and (where relevant) GDPR.

---

📝 Changelog

2026-07-06

• ✨ README overhaul: richer output sample, ready-to-run example scenarios, cross-promo links, and clearer quick-start.

2026-07-01

• Maintenance pass: re-verified end-to-end on live data and confirmed successful runs within the 5-minute quality window on the default input.
• Sharpened Store metadata (SEO title & description) and expanded the FAQ with high-intent, long-tail questions for easier discovery in Google and Apify Store search.
• Added ready-to-run example tasks that cover common real-world use cases.

2026-06-15

• Initial release — extract archived URLs from the Wayback Machine CDX API with date/status filters, CSV/JSON export, no API key.