Wayback Machine URL Recovery & Migration Audit

Recover historical website URLs from the Internet Archive's Wayback Machine CDX index. Build an archived URL inventory, find URLs missing from your current sitemap list, and export structured JSON or CSV for SEO migrations and website research.

What you can do

• Recover unique historical HTML page URLs for a website or folder.
• Prepare a historical URL inventory before a website migration.
• Compare recovered URLs against a current sitemap or exported URL list.
• Follow each result to its archived capture using the provided Wayback link.
• Export clean records through Apify Console or integrate with the API.

The Actor returns index metadata. It does not download archived page bodies, restore websites, check current HTTP status codes, or identify semantic content changes.

Quick start

1. Open the Actor's Input tab in Apify Console.
2. Enter a domain such as example.com, or a folder such as https://docs.python.org/3/.
3. Set your maximum number of URLs, or keep the default of five.
4. Optionally paste the current URLs from your sitemap export.
5. Run the Actor and inspect the Dataset tab. Export JSON or CSV there.

A bare domain or a root website URL includes that hostname and its subdomains. A URL with a path uses that path as a prefix. For example, /docs/ includes pages beneath that folder. Domains with and without www can have different scope; folder queries follow the CDX index's common www normalization. Use a domain without www if you need both the main hostname and its subdomains.

Inputs

Field	Required	Default	Meaning
website	Yes	—	Public domain or folder URL, without query parameters or fragments.
maxResults	No	5	Maximum unique URLs to deliver; between 1 and 10,000.
currentUrls	No	Empty list	Current public HTTP(S) URLs to compare against; leave empty for recovery only.

Only the website is required:

{"website": "example.com"}

Recover a folder and compare it with your supplied current list:

{
  "website": "https://example.com/docs/",
  "maxResults": 100,
  "currentUrls": [
    "https://example.com/docs/",
    "https://example.com/docs/getting-started"
  ]
}

Comparison ignores HTTP versus HTTPS, fragments, hostname capitalization and default ports. Path capitalization and query parameters remain significant. It does not automatically merge unrelated subdomains or rewrite URLs.

Output

Each dataset row describes one unique archived URL. CDX index identities and conservative URL normalization suppress repeated captures across pages and restarts. A capture with a successful historical HTTP status and HTML MIME type supplies the metadata.

Example from the verified live dataset (no current-URL comparison supplied):

{
  "url": "http://docs.python.org:80/3/",
  "domain": "docs.python.org",
  "capturedAt": "2012-10-29T03:46:10Z",
  "archiveUrl": "https://web.archive.org/web/20121029034610/http://docs.python.org:80/3/",
  "archivedStatusCode": 200,
  "mimeType": "text/html",
  "contentDigest": "6ILGSKJ2N36XB5EF7K2TMR75FTS7LRJO",
  "inCurrentUrls": null
}

Field	Meaning
url	Original archived page URL.
domain	Hostname from the original URL.
capturedAt	UTC time of the selected archived capture, not publication time.
archiveUrl	Wayback link for this capture; replay availability can change.
archivedStatusCode	Historical index status; currently only successful 200 captures are returned.
mimeType	HTML or XHTML MIME type.
contentDigest	Archive-provided content fingerprint, or null if unavailable.
inCurrentUrls	True or false against your supplied list; null when no list is supplied.

inCurrentUrls: false means the recovered URL is missing from the list you supplied. It does not establish that the current page is deleted, broken or returning 404.

HTTP failure captures, CDX headers, resume markers, invalid records and duplicates are excluded from the dataset. A stored HTTP 200 is index evidence: without downloading page bodies, the Actor cannot identify soft 404 pages or judge the contents of a URL such as 404.html. The separate Coverage and stop reason output links to a run summary containing delivered counts, rejected records, source failures and whether coverage is partial.

Coverage and recovery

The result limit can stop an export before the archive is exhausted. A successful run may therefore have partial coverage. Check the summary's complete, partial, stopReason, sourceError and hasMoreResults fields before treating an inventory as exhaustive.

If Internet Archive returns an access denial or remains unavailable after bounded retries, the Actor preserves delivered rows and reports the source problem. A source failure with no delivered rows fails the run instead of presenting an empty archive as success. A genuinely empty valid CDX response can complete with zero records.

Apify's resurrection feature can continue a stopped run with its original input and default storage. The Actor reconciles the dataset and drains saved pending rows before requesting another page. Starting a fresh run creates a separate inventory; it does not automatically reuse another run's checkpoint.

hasMoreResults is true when pending rows or a usable continuation remain, false only for verified complete coverage, and null when more results cannot be established. recoveryAdvice explains the appropriate next step. A result-limit stop needs a new run with a larger maxResults; resurrection keeps the original input limit. A spending-limit stop needs a budget review: resurrection does not automatically increase the original cap.

Graceful interruption writes the summary and URL-quality report. Hard platform termination can prevent final reports from being refreshed; after resurrection, delivered records are reconciled before continuing. Compare a report's analyzed count with the dataset count when inspecting a stopped run.

Advisory URL quality

The Advisory URL quality output links to a separate, uncharged URL_QUALITY report. Dataset rows retain the same eight fields and their original URLs. The report contains analyzedUrlCount, flaggedUrlCount, per-flag counts, and entries with each flagged url and its flags.

• tracking_query: query keys beginning with utm_, or _ga, _gl, gclid, fbclid or msclkid (case insensitive).
• error_like_path: a final path name of 404, 500, error, not-found or not_found, ignoring its file extension.

These are advisory URL patterns. A legitimate research page named 404.html may be flagged; the report cannot establish spam, soft 404s or page content. No flagged URL is automatically removed or rewritten. Rejected, pending and duplicate rows do not add warnings. Reports are reconstructed from delivered records after recovery and written at completion or graceful interruption, without a report write for every URL.

Pricing

Pricing is $0.0171 per 1,000 delivered URLs ($0.0000171 per URL), plus $0.0032 per run start until 23 October 2026 and $0.0061 per run start from 23 October 2026 (00:00 UTC) at the fixed 256 MB allocation. Platform usage is included. Before that change, a 100-URL run costs $0.00491 in events and a 1,000-URL run costs $0.0203. From 23 October, those totals are $0.00781 and $0.0232 respectively. The higher startup rate covers the measured cost of bounded upstream timeouts. See the Pricing tab for the effective configuration.

One validated unique dataset row triggers one URL event. Retries, rejected records, duplicates and status summaries are not URL events. The startup event applies even when the archive is empty or unavailable. A fresh run and each resurrection incur another startup event. Resurrection preserves the original dataset and URL-event counts; already delivered URLs are not emitted again.

See the Pricing tab for the current event rates before starting a run.

Use a positive Apify maximum spending limit to cap a run; zero is treated by the platform as no explicit limit. Owner tests can also consume platform usage within their account budget. When the event budget runs out, the Actor stops promptly and preserves pending rows. A result-count limit controls inventory size; it does not establish a spending limit across multiple runs.

Python API example

Install the official apify-client package and provide your Apify token through your application's environment.

import os
from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("thescrapelab/wayback-machine-url-recovery").call(
    run_input={"website": "https://docs.python.org/3/", "maxResults": 100},
    max_total_charge_usd=Decimal("0.25"),
)
if run is None:
    raise RuntimeError("Actor call returned no run or storage metadata")
if run is None or (run["status"] if isinstance(run, dict) else run.status) != "SUCCEEDED":
    raise RuntimeError("Check the Actor run log and coverage summary")

for item in client.dataset((run["defaultDatasetId"] if isinstance(run, dict) else run.default_dataset_id)).iterate_items():
    print(item["url"], item["archiveUrl"])

summary = client.key_value_store((run["defaultKeyValueStoreId"] if isinstance(run, dict) else run.default_key_value_store_id)).get_record("RUN_SUMMARY")
print(summary["value"] if summary is not None else "No summary available")
quality = client.key_value_store((run["defaultKeyValueStoreId"] if isinstance(run, dict) else run.default_key_value_store_id)).get_record("URL_QUALITY")
print(quality["value"] if quality is not None else "No quality report available")

This example uses Apify Python client 3.x. Run metadata uses typed objects; dataset rows and key-value records are dictionaries. Report contents are under value.

Limitations and responsible use

• The Wayback Machine does not archive every website or every page. Missing index results do not prove that a URL never existed.
• Captures can be excluded, removed or unavailable for replay. The Actor does not bypass exclusions or restricted access.
• This product currently inventories successful HTML pages. Images, JavaScript, PDFs, failed captures and archived page contents are outside its output.
• A digest is archival metadata; it does not identify a price change, policy change or publication date.
• Internet Archive is an independent upstream service. Its availability and response times affect runs; each run makes sequential CDX requests at least three seconds apart, uses bounded retries, and honors Retry-After. Run multiple website jobs sequentially to avoid multiplying the source request rate.
• Use archive metadata responsibly and respect source restrictions. Do not collect or store personal data about anyone.
• Archived links do not grant rights to reuse the underlying content. Review Internet Archive's terms and applicable rights for your intended use.

The source is the Internet Archive CDX API. This Actor is independently developed by TheScrapeLab and is not affiliated with or endorsed by Internet Archive.

Support

Use the Actor's Issues tab and include the run ID, website scope and the summary's stop reason. Never include API tokens or private credentials.