Wayback Machine URL Extractor – Archived URLs, Snapshots & Diff

List every URL the Internet Archive's Wayback Machine has saved for a domain, subdomain or URL prefix. You get one clean row per unique URL with the first and last capture date, capture count, number of content versions, last HTTP status, MIME type and direct snapshot links. Optionally the Actor downloads the archived page as Markdown or text, or compares a page between two dates (similarity, lines added and removed, unified diff).

It uses the official Wayback CDX API, with no browser and no proxy, so it is fast and costs $0.20 per 1,000 URLs.

What it does

• All archived URLs of a site. Give example.com (domain + subdomains), example.com/blog/ (URL prefix) or an exact page URL. The CDX API results are de-duplicated by urlkey, so http://, https://, www. and :80 variants of the same page become one row.
• Capture statistics per URL: firstCapture, lastCapture, captureCount, uniqueVersions (distinct content digests), lastOkCapture (last HTTP 200), the latest status code, MIME type and size.
• Snapshot links: latestSnapshotUrl (normal Wayback view), latestRawSnapshotUrl (id_ raw archived bytes, no toolbar) and firstSnapshotUrl.
• Filters on the Internet Archive server, so nothing filtered is downloaded or charged:
  • date range (from / to: 2015, 2019-06-30, 2 years),
  • HTTP status (200, 3xx, 404),
  • MIME type (text/html, application/pdf, image/*),
  • URL regex include and exclude,
  • skip query-string URLs.
• Clean by default. Tracking-parameter variants (utm_*, fbclid, gclid, __hstc, session IDs) and malformed URLs harvested from broken HTML ("><span>, %22, glued URLs) are dropped for free.
• Snapshot content (optional). The latest (or earliest) successful capture is downloaded in raw id_ mode and converted to clean Markdown, plain text or raw HTML. This is charged per snapshot.
• Diff between two dates (optional). For each URL, the HTTP 200 capture closest to each date is downloaded, converted to Markdown and compared. You get changed, similarity, linesAdded, linesRemoved, addedLines, removedLines and a unifiedDiff.
• Polite and robust. One CDX request at a time with at least 1 s between calls, automatic slow-down and exponential backoff on HTTP 429/5xx (honouring Retry-After), an honest User-Agent with a contact address, and a safe stop if the archive keeps rate-limiting. Large result sets are paged with resumeKey. Very busy URLs (home pages with 100,000+ captures) are counted up to maxCapturesPerUrl and then skipped ahead, so the run never stalls on one URL.

Use cases

• SEO and site migrations: find every old URL of a domain to build a 301 redirect map, recover lost pages and backlinks, and audit legacy URL structures.
• Expired domain research: see what a domain hosted, when, and how often it was crawled before you buy it.
• Content recovery: get deleted blog posts or docs back as Markdown from their last good capture.
• Competitive intelligence: track how a competitor's pricing, terms or landing pages changed between two dates.
• Security and bug bounty recon: list historical endpoints, parameters, .js, .json, .pdf and backup files (like waybackurls and gau, as a hosted API).
• Research, journalism and compliance: prove what a page said on a given date, with the archive link.
• AI and RAG: feed archived page text into LLMs for historical analysis.

Input example

{
  "urls": ["example.com", "example.org/blog/", "https://example.net/pricing"],
  "matchType": "auto",
  "maxUrls": 5000,
  "from": "2018",
  "to": "2024-12-31",
  "statusCodes": ["200"],
  "mimeTypes": ["text/html"],
  "excludeUrlRegex": ".*\\.(css|js|png|jpg|gif|svg|woff2?)",
  "captureStats": true,
  "fetchContent": false,
  "compareFrom": "",
  "compareTo": ""
}

The default input lists 200 archived URLs of crawlee.dev in about 3 seconds.

Field	Default	Notes
urls	crawlee.dev	Domains, *.domain, URL prefixes (/ or * at the end) or exact URLs
matchType	auto	domain / host / prefix / exact force one type for all entries
maxUrls / maxUrlsPerInput	200 / 0	0 = no limit
from / to	–	2015, 2015-06, 2015-06-30, or relative 6 months
statusCodes / mimeTypes	–	e.g. 200, 3xx / text/html, image/*
urlRegex / excludeUrlRegex	–	Java regex that must match the whole original URL
skipQueryStrings	false	Drop every URL with ?
skipTrackingUrls	true	Drop utm_*, fbclid, gclid, session-ID variants only
skipMalformedUrls	true	Drop broken-HTML junk URLs
captureStats	true	Off: one CDX row per URL (faster on huge domains, only the first capture is known)
maxCapturesPerUrl	5000	Counting cap for very busy URLs
fetchContent, contentFormat, contentSnapshot	false, markdown, latest	Archived page text
compareFrom / compareTo	–	Set both to get a diff per URL

Output example

One dataset row per unique archived URL. The Archived URLs, Snapshot content, Diff between two dates and Errors views are in the Output tab.

{
  "type": "url",
  "url": "https://apify.com/pricing",
  "host": "apify.com",
  "path": "/pricing",
  "extension": null,
  "urlkey": "com,apify)/pricing",
  "firstCapture": "2017-10-27T07:20:08Z",
  "lastCapture": "2026-10-03T03:43:41Z",
  "captureCount": 181,
  "captureCountCapped": false,
  "uniqueVersions": 176,
  "lastOkCapture": "2026-10-03T03:43:41Z",
  "mimeType": "text/html",
  "statusCode": 200,
  "lengthBytes": 55928,
  "firstSnapshotUrl": "https://web.archive.org/web/20171027072008/https://www.apify.com/pricing",
  "latestSnapshotUrl": "https://web.archive.org/web/20261003034341/https://apify.com/pricing",
  "latestRawSnapshotUrl": "https://web.archive.org/web/20261003034341id_/https://apify.com/pricing",
  "input": "https://apify.com/pricing",
  "matchType": "exact",
  "scrapedAt": "2026-10-03T09:46:40Z"
}

• With fetchContent, a row also has contentStatus (ok / failed / skipped), contentSnapshotDate, contentSnapshotUrl, title, contentWordCount, content, contentTruncated and contentError.
• With compareFrom and compareTo, a row also has compareFromSnapshot, compareToSnapshot (the actual capture dates used), sameCapture, changed, similarity (0–1), linesAdded, linesRemoved, addedLines, removedLines, unifiedDiff, titleBefore/titleAfter, diffStatus and diffError.
• Inputs with no captures, invalid inputs and refused queries become rows with "type": "error", which are free.
• The key-value store record OUTPUT holds the run summary: per input, the URLs saved, captures scanned, CDX requests, malformed URLs skipped and why the run stopped.

See SAMPLE_OUTPUT.json for full rows with content and diff.

Pricing

Pay per event. You pay only for URLs saved and snapshots downloaded.

Event	Price
Archived URL saved	$0.0002 ($0.20 / 1,000)
Snapshot fetched (content or diff, optional)	$0.001 ($1 / 1,000)
Actor start	$0.00005

A diff uses up to 2 snapshots per URL. If both dates resolve to the same capture, it is downloaded and charged only once. Non-text captures (images, PDFs) and failed downloads are not charged.

Compared with other Store Actors (public Store prices, 2026-10-03):

Actor	Price per 1,000 URLs
Wayback Machine URL Extractor (this Actor)	$0.20
andok (Wayback)	$1.00
ryanclinton/wayback-machine-search	$3.50
logiover (Wayback)	$5.00

Set Maximum cost per run on the run options to cap spending. The Actor stops cleanly when the limit is reached and never downloads a snapshot it cannot charge for.

FAQ

Why do I get fewer URLs than tools like waybackurls? By default, rows are unique pages (urlkey). Tracking-parameter variants and malformed URLs are dropped. Set skipTrackingUrls and skipMalformedUrls to false to get the raw list.

Some big sites return an error. For a few very large sites (for example nytimes.com and reddit.com), the Internet Archive refuses anonymous domain and prefix listings ("This type of CDX query requires authorization"). The Actor reports this as an error row, free of charge. Exact page URLs on those sites still work, including content and diff.

What does captureCountCapped mean? Home pages can have hundreds of thousands of captures. After maxCapturesPerUrl captures (default 5,000), counting stops for that URL, the newest capture is looked up directly, and the scan moves on. Raise the limit for exact counts.

Do filters change the statistics? Yes. firstCapture, lastCapture and captureCount count only the captures that match your date, status and MIME filters. For example, with statusCodes: ["200"], captureCount is the number of successful captures.

How is "latest snapshot" chosen? latestSnapshotUrl and the content download use the newest HTTP 200 capture (lastOkCapture), so you do not land on an archived 404 or redirect. lastCapture and statusCode describe the newest capture of any status.

How fast is it? Listing runs at about 10,000 captures per CDX request, at most one request per second. With captureStats off, one CDX row is one URL, so huge domains list quickly. Snapshot downloads run 2 at a time with at least 0.5 s between requests, out of respect for the Internet Archive's free service.

Is it legal? The Actor uses the Internet Archive's public CDX and Wayback APIs at a polite rate, identifies itself, and does not bypass any exclusion. The archived content belongs to its original owners, so check copyright before you republish it. If you use the Internet Archive heavily, please consider donating to it.

Use with AI agents / Apify MCP

• Apify MCP server: add gazidev/wayback-url-extractor to your MCP client (Claude Desktop, Cursor, VS Code) via https://mcp.apify.com?actors=gazidev/wayback-url-extractor. An agent can call it with {"urls":["https://competitor.com/pricing"],"compareFrom":"1 year","compareTo":"1 day"} and get back what changed on the page.
• API: POST https://api.apify.com/v2/acts/gazidev~wayback-url-extractor/run-sync-get-dataset-items?token=... with the input JSON returns the rows directly.
• Scheduled monitoring: a weekly Task with compareFrom: "7 days" and compareTo: "1 day" on key competitor pages can feed the Slack, Google Sheets or webhook integration.

More from the website toolkit

• Sitemap Extractor: every URL a site lists in its sitemap.xml today. Diff it against this Actor's output to find removed pages.
• Website to Markdown: any live URL, sitemap or whole site to clean Markdown with RAG chunks and llms.txt.
• SEO Audit Crawler: broken links, meta tags and redirects.
• RSS Feed Reader: any RSS, Atom or JSON Feed to JSON, with only-new mode.
• Tech Stack Detector: bulk Wappalyzer alternative.
• Domain Checker: WHOIS/RDAP, DNS, SSL and email security in bulk.
• Website Contact Finder: emails, phones and social links from websites.

Not affiliated with the Internet Archive. Wayback Machine is a trademark of the Internet Archive.