Wayback Machine Scraper - Snapshots & Every Archived URL

Useful for recovering deleted pages, building redirect maps after a migration, auditing old subdomains, and checking what an expired domain used to host: look up what the Internet Archive's Wayback Machine holds for any page or domain, straight from its official CDX API.

It does two things:

• Snapshots — one row per archived capture of a URL: when it was captured, the HTTP status the page returned, the content type and size, whether the content changed since the capture before, and a link to the archived copy (plus a link to the raw archived file with no Wayback toolbar).
• Every archived URL — one row per distinct URL the archive holds under a domain, host or path: when it was first and last captured, how many captures there are, every status code it has returned, and the latest status. In this mode every row already carries statusCodesSeen, so pages that ever returned a 404 can be picked straight out of the list.

The archive's own API falls over on exactly the domains you care about, and all of it was measured on 2026-09-10 and 2026-09-11. Asking for every unique URL under a domain the archive's own way can time out with HTTP 504 after 60 seconds, even for 20 URLs (2026-09-10) — so captures are read here in light 5,000-row pages and grouped locally, which is why large domains work. With no limit, one page alone can return 768,693 rows, so every request carries a limit. And the archive's own paging silently skips captures at the page boundary: at apify.com/ on 2019-07-11 22:42:57 there are two captures, a 200 and a 301, and paging one at a time returns only the 200.

Nothing is dressed up as data it is not. About one capture in eight is a "revisit" with status - (377 of 3,000 captures under apify.com): statusCode comes back null with captureType: "revisit", never a fake 0. 503 and 504 are routine, so each request is retried with increasing waits — and if it still fails, a row says so instead of reporting "no captures". google.com answered 504, 504, 504, 503, 504 over 285 seconds on 2026-09-11, and that is reported as what it was. nytimes.com refuses URL listings outright with HTTP 403 whatever the date range, and that is reported too.

The status filter is sent to the archive rather than applied afterwards, because filtering after thinning out drops whole months — 5 of apify.com's latest 24 monthly captures are 301 redirects.

No API key, no browser, no proxy. Export as JSON, CSV or Excel.

What the archive gets wrong, and what this Actor does about it

These were measured against the live CDX API on 2026-09-10.

The archive's behaviour	What you get here
A mistyped domain and a broken date both come back as the same empty answer as a real page that was never archived	Entries are checked before they are sent. A malformed URL or impossible date is returned as its own row explaining why, never as "no captures"
A malformed URL such as http://[bad is silently answered with captures of a different address (bad)	Host names are validated; entries that are not a real domain or IPv4 address are not sent
About one capture in eight is a "revisit" with status - (377 of 3,000 captures under apify.com)	statusCode is null and captureType is revisit - never a fake 0
Content types include unk and unknown, which are not content types	mimeType is null for those
Asking for every unique URL under a domain the archive's own way can time out (HTTP 504 after 60 seconds, even for 20 URLs on 2026-09-10)	Captures are read in light 5,000-row pages and grouped here, so large domains work
With no limit, one page alone can return 768,693 rows	Every request carries a limit
503 "Temporarily Offline" and 504 timeouts are routine	Each request is retried with increasing waits; if it still fails, a row says so instead of reporting "no captures"
The most heavily archived pages overwhelm the archive's own servers. For google.com, asking for one capture per year timed out (HTTP 504 after 60 seconds) oldest-first and newest-first, and even single-year lookups failed 4 times in 7	Each request is retried with increasing waits for up to about eight minutes (google.com took 285 seconds on 2026-09-11), and for less when the run is close to its time limit. If the archive still cannot answer, you get a failed row saying so - nothing is guessed. Pages with a normal amount of history (apify.com, python.org, docs.python.org, github.com, openai.com) answered in 3 to 25 seconds in the same tests
A run has a time limit (one hour by default), and those heavily archived pages take about five minutes each: google.com answered HTTP 504, 504, 504, 503, 504 over 285 seconds on 2026-09-11. Many of them in one run would reach the limit and the platform would cut the run off, leaving the remaining entries with no row at all	The Actor knows when the run will end. Each request's wait and the retries are shortened so that no lookup runs past that point, and when less than 75 seconds are left it stops querying and every remaining entry gets its own skipped-time-limit row. An entry whose retries were cut short this way gets the same status, and a URL list cut short part-way ends with an incomplete row
Some sites are refused for URL listings: nytimes.com answers every domain, host and path listing with HTTP 403 "This type of CDX query requires authorization", whatever the date range (apify.com, python.org and wikipedia.org list normally)	A refused-by-archive row says the archive declined, that retrying will not help, and that single-page snapshots of that site still work - it is not passed off as a temporary error or as "no captures"
The archive's own paging (its resume key) silently skips captures at the page boundary: at apify.com/ on 2019-07-11 22:42:57 there are two captures, a 200 and a 301, and paging one at a time returns only the 200	Snapshots are read in one request (up to 10,000 rows), so nothing is paged. URL lists re-ask for the boundary second of every page and add what the archive skipped, without counting anything twice
Filtering by status after thinning out drops whole months: 5 of apify.com's latest 24 monthly captures are 301 redirects	The status filter is sent to the archive, which applies it before thinning out - the same 24 months all come back with a 200

If a URL list stops early (a page of the archive could not be read, the reading limit was reached, or the run was close to its time limit), every URL except the last one is still fully counted - the archive returns captures grouped by URL in date order - and that last one is marked complete: false, with a row explaining that the list is incomplete. A URL whose page-boundary re-check failed is also marked complete: false.

Addresses with a non-standard port (such as :8080) are rejected rather than looked up, because dropping the port would return captures of a different address.

Input

Field	Default	What it does
urls	-	Pages or domains. In URL-list mode, *.example.com means every subdomain and example.com/blog/* means everything under that path
mode	snapshots	snapshots or urls
matchType	domain	URL list only: domain (host and subdomains), host, or prefix
from / to	-	YYYY, YYYY-MM, YYYY-MM-DD or the archive's own digits (20210315). Inclusive
collapse	month	Snapshots only: one per month, day or year, only when the content changed, or none
order	newest	Snapshots only: newest or oldest first
statusFilter	any	Snapshots only: ok (2xx), redirect (3xx) or error (4xx/5xx). The archive applies it before thinning out, so with collapse: month you get one matching capture per month rather than losing the months whose first capture was a redirect. Revisits carry no status and only appear with any
maxResultsPerUrl	100	Up to 10,000 rows per entry

If urls is empty, apify.com is looked up so the Actor always returns something, and every row records what was looked up in its input column.

Examples

Monthly snapshots of a page over two years:

{ "urls": ["https://www.nytimes.com/"], "from": "2023", "to": "2024", "collapse": "month", "order": "oldest" }

Every page ever archived under a site's blog:

{ "urls": ["example.com/blog/*"], "mode": "urls", "maxResultsPerUrl": 2000 }

Captures of one page that recorded an error (4xx or 5xx), newest first:

{ "urls": ["example.com"], "statusFilter": "error", "collapse": "none" }

In URL-list mode every URL row already carries statusCodesSeen, so pages that ever returned a 404 can be picked out of the list directly.

Output

Snapshot row:

{
  "rowType": "snapshot",
  "input": "apify.com",
  "url": "https://apify.com/",
  "capturedAt": "2026-09-07T22:14:23Z",
  "timestamp": "20260907221423",
  "statusCode": 200,
  "statusClass": "ok",
  "captureType": "capture",
  "mimeType": "text/html",
  "lengthBytes": 73407,
  "digest": "FLYN6YGUV5ZLNCIBCXESIGJ43HLXV6NL",
  "sameContentAsPrevious": false,
  "archiveUrl": "https://web.archive.org/web/20260907221423/https://apify.com/",
  "rawArchiveUrl": "https://web.archive.org/web/20260907221423id_/https://apify.com/"
}

URL-list row (a real row, docs.apify.com with scope host, 2026-09-10):

{
  "rowType": "url",
  "input": "docs.apify.com",
  "url": "https://docs.apify.com/",
  "urlKey": "com,apify,docs)/",
  "firstCapturedAt": "2019-12-11T00:41:01Z",
  "lastCapturedAt": "2026-08-21T23:05:38Z",
  "captureCount": 200,
  "statusCodesSeen": [200, 301, 403],
  "statusCode": 200,
  "mimeType": "text/html",
  "archiveUrl": "https://web.archive.org/web/20260821230538/https://docs.apify.com/",
  "firstArchiveUrl": "https://web.archive.org/web/20191211004101/https://docs.apify.com/",
  "complete": true
}

The archive also holds malformed addresses its crawler picked up - on docs.apify.com, for example, paths with fragments of JSON stuck to them. They are returned exactly as the archive records them, usually with a 404 in statusCodesSeen, so they are easy to filter out.

lengthBytes is the size of the stored (compressed) capture, as the archive reports it. sameContentAsPrevious compares the capture with the one immediately before it in time; with collapse: month that means the previous month's capture.

Rows with status other than ok (no-captures, no-matching-status, bad-input, refused-by-archive, failed, incomplete, skipped-time-limit) explain themselves in note and are never charged. skipped-time-limit means the entry was not answered because the run was about to reach its time limit (it was not looked up, or its retries were cut short) - run it again or give the run a longer timeout.

Which capture represents a period. With one per month, day or year, newest-first order returns the latest capture of each period and oldest-first the earliest (for apify.com in June 2026: 06-28 newest-first, 06-07 oldest-first). "Only when the content changed" always returns the capture where the change happened: in newest-first order the archive itself would return the last capture of each unchanged run, so this Actor reads up to the newest 10,000 captures un-thinned and picks the change points itself.

Pricing

Pay per result: you are charged for each snapshot row or URL row returned. Rows that only explain an empty, rejected or failed lookup are free.

Source

Data comes from the Internet Archive's public Wayback CDX server (web.archive.org/cdx/search/cdx). This Actor is not affiliated with the Internet Archive.