Wayback URL Finder: Lost Pages, History & Redirect Gaps

Urlwright. Little tools for the web.

Old URLs of any domain, checked today for redirect gaps. The actor reads the public Wayback Machine and, when you ask it to, requests those URLs on the live web. You get metadata: the URL, when it was seen, the archived status, the MIME type, a content digest, and a link to the capture. Page bodies are never stored.

Uses the public Wayback Machine CDX API of the Internet Archive. Not affiliated with or endorsed by the Internet Archive.

What it does

1. You pass domains (example.com) or URL prefixes (https://example.com/blog/). A dataset from Sitemap URL Extractor, Tech Stack Detector or Brand Kit can supply more targets.
2. Lost URLs lists each archived URL once, with the first and last time it was captured, how many snapshots were collapsed, the archived status, MIME type and digest, and links to the first and last capture.
3. Domain history returns one row per target: first and last capture, captures per year, years with no captures, and a sample of status and MIME types.
4. URL lookup returns the capture closest to a date you choose.
5. Optional live check: HEAD, then GET if the server rejects HEAD. Redirects are followed (at most five). robots.txt for UrlwrightBot is respected unless you turn that off.
6. Optional sitemap comparison. Pass a Sitemap URL Extractor dataset and each URL is marked inSitemap. A redirect gap is an archived HTML page that today returns 404 or 410, was not redirected, and (when you passed a sitemap) is not in that sitemap.
7. URLs that contain an email address, or a session, token or password parameter, are dropped and counted. They are not written out.

One target that the archive cannot answer does not fail the run. RUN_SUMMARY is always written to the key value store, including when the input is rejected.

Who it is for

• Relaunches: find URLs the new site forgot, so backlinks do not end on a 404.
• Buying a domain: see the years it was captured, and the years it was not.
• Showing when a page was public: the row carries archive links. Evidential value must be checked by a lawyer.

Input

The console prefills a small run (https://urlwright.com, 25 URLs, 10 live checks). Defaults for a real run are higher: 1,000 URLs per target and 200 live checks.

{
  "startUrls": ["https://urlwright.com", "https://example.com/blog/"],
  "mode": "lostUrls",
  "includeSubdomains": false,
  "from": "2018",
  "to": "2024",
  "mimeTypes": ["text/html"],
  "statusCodes": ["200"],
  "maxUrlsPerTarget": 1000,
  "checkLiveStatus": true,
  "maxLiveChecksPerTarget": 200,
  "respectRobotsTxt": true,
  "sitemapDatasetId": ""
}

datasetId plus datasetField (default url) reads extra targets. If the named field is missing on a row, the actor tries url, then finalUrl, then domain, then inputUrl.

includeSubdomains false asks for a domain once, with matchType=domain filtered to the apex host and www. True queries the host and its subdomains. A path is still queried on the host and its www twin, because a domain query is not a path prefix. A host that already has a subdomain label, such as blog.example.com, is queried as written.

Dates are YYYY, YYYY-MM or YYYY-MM-DD. from must be on or before to. lookupDate is required when mode is lookup.

Output

One dataset, two row types. Every field is present. Unused fields are null.

An archived URL:

{
  "schemaVersion": 1,
  "type": "archived-url",
  "inputUrl": "https://example.com",
  "domain": "example.com",
  "url": "http://example.com/about",
  "status": "ok",
  "httpStatus": 404,
  "finalUrl": "http://example.com/about",
  "redirectChain": [],
  "archiveHttpStatus": 200,
  "mime": "text/html",
  "digest": "ABCDEF",
  "firstSeen": "2016-03-01T00:00:00Z",
  "lastSeen": "2019-11-02T00:00:00Z",
  "snapshotCount": 4,
  "firstArchiveUrl": "https://web.archive.org/web/20160301000000/http://example.com/about",
  "lastArchiveUrl": "https://web.archive.org/web/20191102000000/http://example.com/about",
  "inSitemap": false,
  "redirectGap": true,
  "charged": true,
  "chargedEvents": { "archived-url": 1, "domain-history": 0, "live-status-checked": 1 },
  "error": null
}

mime is the MIME type of the archived capture. httpStatus is the status today. snapshotCount is null when the archive does not return a group count. Views: Overview, Redirect gaps, History.

Live checks are the only requests sent to a customer host. CDX requests go only to web.archive.org. When the SSRF guard blocks a live check, on the first request or on any redirect, error is ssrf-blocked, httpStatus is null and redirectGap is null. That live check is not billed. The archived-url charge on the same row still applies, because the archive metadata was delivered.

RUN_SUMMARY counts rows, malformed archive lines, filtered personal URLs, and stoppedReason (charge-limit or timeout) when the run stops early. REDIRECT_GAPS.csv is written when at least one gap was found.

A domain history row uses type: "domain-history" and status ok, partial, no-captures, archive-unavailable or invalid. distributionSample is true: the status and MIME counts come from collapsed archive rows, not from every capture. partial is true, and status is partial, when a query was capped or failed. That row is free. Only status: ok is billed.

Pricing

Pay per event. No start fee. A failed archive query, a duplicate row and a filtered personal URL are free. A live check blocked by the SSRF guard is reported as blocked and is not billed. The archived URL on that row is still billed, because the archive metadata was delivered. Prices in USD. PLATINUM and DIAMOND use the GOLD price. The minimum max charge per run is $0.02.

Event	What you pay for	FREE	BRONZE	SILVER	GOLD
archived-url	One archived URL written to the dataset	0.0005	0.00045	0.0004	0.0003
domain-history	One domain history with status ok	0.006	0.0055	0.0052	0.005
live-status-checked	One live check that received an HTTP status	0.0005	0.00045	0.0004	0.0003

A failed or unavailable archive query is never billed. Small runs usually finish in 30 to 120 seconds.

At the FREE price, 1,000 archived URLs are $0.50. The same 1,000 rows with a live status are $1.00. A domain history is $0.006.

Limits and runtime

The Internet Archive asks callers to slow down. This actor sends at most 0.8 CDX requests per second, one at a time, and does not rotate IPs. A page asks for at most 2,000 collapsed URLs, and only as many as the run can still keep, plus a small buffer. 1,000 URLs are usually one request: a few seconds when the archive is quick, plus 1.25 seconds between requests. Larger lists add at least 1.25 seconds per extra page. Saved rows are written and billed in batches, so a long list does not wait on a separate platform call for every URL. Each CDX attempt waits at most 30 seconds, and one target stops after three attempts or 120 seconds of archive time, whichever comes first. Live checks add about one second per request on a single host, with up to four hosts at once.

The archive can be down or slow. A target then comes back as archive-unavailable, with the last HTTP status or timeout in error, and is not billed. A history that is only partly filled comes back as partial and is not billed. Personal URLs are removed before you see them.

Suggested run options: build latest, 128 MB memory, timeout 3600 seconds. Memory can be raised up to 1024 MB. Measured runs peaked at 31 to 45 MB.

FAQ

Does it download old pages? No. Metadata and archive links only.

Can I use this as proof in a dispute? The archive links show what the Wayback Machine stored. Evidential value must be checked by a lawyer.

Why are some URLs missing? The archive may not have captured them, the MIME type or status filter excluded them, or a personal parameter caused them to be dropped. RUN_SUMMARY reports the counts.

What is a redirect gap? Archived HTML that returned a 2xx status, and today returns 404 or 410 without a redirect. If you passed a sitemap dataset, the URL also has to be absent from that sitemap.

Will a failed target fail the run? No. Other targets continue. The summary explains each one.

No warranty

The output depends on the Internet Archive and on the sites you check. Captures can be missing, incomplete or wrong. Nothing in the dataset is legal advice, and no warranty is given that a URL was public, or that a live status will stay the same.