Wayback Lost Pages Finder

Find the pages your site used to have that are gone today. The Actor asks the Internet Archive's Wayback Machine for every HTML page it ever archived on a domain, then checks each one on the live site and tells you what happens to it now:

Status	Meaning
lost	The old URL returns 404 or 410. Backlinks and bookmarks to it are dead
redirected-to-homepage	The old URL redirects to the homepage. Google usually treats this as a soft 404
redirected	The old URL redirects to another page that loads
live	The page still loads at the same address (an http → https or www switch counts as the same address)
error	5xx, other 4xx, timeout, redirect loop or too many hops
not-checked	robots.txt disallows the URL, so it was not requested (not charged)

Use it after a migration, a redesign, a CMS change or a domain move to find the URLs nobody put in the redirect map, or before you buy or relaunch a domain to see what it used to contain.

Input

• Domains: one or more domains, e.g. example.com. www. is folded into the bare domain.
• Max pages per domain (default 500): the cap on unique archived pages checked.
• Include subdomains: also list blog.example.com, shop.example.com and so on.
• Archived from / up to year: restrict to captures from a period, e.g. up to the year before your migration.
• Include URLs with query strings: off by default, because archived ?utm= and session variants are usually duplicates.
• Skip file extensions: the defaults drop assets (css, js, images, fonts, pdf, office files, media).

Output

One dataset row per archived page:

{
  "domain": "crawlee.dev",
  "archivedUrl": "https://crawlee.dev/api/3.0/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "firstArchived": "2023-06-06T07:10:04Z",
  "firstSnapshotUrl": "https://web.archive.org/web/20230606071004/https://crawlee.dev/api/3.0/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "latestSnapshotUrl": "https://web.archive.org/web/https://crawlee.dev/api/3.0/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "status": "lost",
  "finalUrl": "https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "finalStatus": 404,
  "hops": 2,
  "note": null,
  "checked": true,
  "checkedAt": "2026-10-28T09:50:14.863Z"
}

finalUrl is where the redirect chain ends, so a lost row with hops > 0 is a redirect that points at a missing page.

The key-value store also has:

• LOST_URLS: the lost URLs as plain text, one per line. Add a target to each and check the result with our Redirect Map Validator.
• OUTPUT: per-domain counts for each status, and whether the archive list was complete or hit your page cap.

How it works

1. It queries the Wayback Machine CDX API (the Internet Archive's public index) for captures that returned HTTP 200 with type text/html, one entry per unique URL. Requests are paced at about one per 1.5 seconds, with back-off on errors.
2. Duplicates are merged: http/https, www/non-www and a trailing slash count as one page.
3. Each page is requested on the live site. Redirects are followed one hop at a time (up to 10). No page bodies are downloaded, only status codes and headers.

How it behaves on your site

• It identifies itself as WaybackLostPages/1.0 and obeys robots.txt for that agent (or *), including Crawl-delay up to 10 s per host. Turn Respect robots.txt off only for sites you own or are authorised to test.
• It collects no personal data. It records URLs, status codes and capture dates.

Pricing

Pay per event: each archived page checked on the live site is one page-checked event. Pages skipped because of robots.txt are not charged. Use Max pages per domain and the run's maximum cost setting to cap spend. The run stops cleanly when the cap is reached.

Limitations

• The Wayback Machine only knows pages it crawled. Small or new sites may have few captures, and pages blocked from the archive are missing.
• Big domains have hundreds of thousands of archived URLs. The list is read in archive order (by URL path) until your cap is reached, so use a year range or a higher cap for full coverage.
• A page that returns 200 with "not found" content (a soft 404) shows as live.
• If the Wayback Machine is down or rate-limits the run, that domain is reported with an error in OUTPUT and nothing is charged for it.

Support

Open an issue on the Actor's Issues tab. Breakages are fixed within 48 hours.

This Actor is built and maintained by an AI agent working for its publisher. A human owns the account and is responsible for it. It is not affiliated with the Internet Archive.