# Wayback Machine URL Recovery & Migration Audit (`thescrapelab/wayback-machine-url-recovery`) Actor

Recover historical website URLs with the Internet Archive CDX API. Compare your current URL list and export clean JSON or CSV for SEO migrations.

- **URL**: https://apify.com/thescrapelab/wayback-machine-url-recovery.md
- **Developed by:** [Inus Grobler](https://apify.com/thescrapelab) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 78.6% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.02 / 1,000 recovered urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Machine URL Recovery & Migration Audit

Recover historical website URLs from the Internet Archive's Wayback Machine CDX index. Build an archived URL inventory, find URLs missing from your current sitemap list, and export structured JSON or CSV for SEO migrations and website research.

### What you can do

- Recover unique historical HTML page URLs for a website or folder.
- Prepare a historical URL inventory before a website migration.
- Compare recovered URLs against a current sitemap or exported URL list.
- Follow each result to its archived capture using the provided Wayback link.
- Export clean records through Apify Console or integrate with the API.

The Actor returns index metadata. It does not download archived page bodies, restore websites, check current HTTP status codes, or identify semantic content changes.

### Quick start

1. Open the Actor's Input tab in Apify Console.
2. Enter a domain such as `example.com`, or a folder such as `https://docs.python.org/3/`.
3. Set your maximum number of URLs, or keep the default of five.
4. Optionally paste the current URLs from your sitemap export.
5. Run the Actor and inspect the Dataset tab. Export JSON or CSV there.

A bare domain or a root website URL includes that hostname and its subdomains. A URL with a path uses that path as a prefix. For example, `/docs/` includes pages beneath that folder. Domains with and without `www` can have different scope; folder queries follow the CDX index's common `www` normalization. Use a domain without `www` if you need both the main hostname and its subdomains.

### Inputs

| Field | Required | Default | Meaning |
| --- | --- | --- | --- |
| `website` | Yes | — | Public domain or folder URL, without query parameters or fragments. |
| `maxResults` | No | 5 | Maximum unique URLs to deliver; between 1 and 10,000. |
| `currentUrls` | No | Empty list | Current public HTTP(S) URLs to compare against; leave empty for recovery only. |

Only the website is required:

```json
{"website": "example.com"}
```

Recover a folder and compare it with your supplied current list:

```json
{
  "website": "https://example.com/docs/",
  "maxResults": 100,
  "currentUrls": [
    "https://example.com/docs/",
    "https://example.com/docs/getting-started"
  ]
}
```

Comparison ignores HTTP versus HTTPS, fragments, hostname capitalization and default ports. Path capitalization and query parameters remain significant. It does not automatically merge unrelated subdomains or rewrite URLs.

### Output

Each dataset row describes one unique archived URL. CDX index identities and conservative URL normalization suppress repeated captures across pages and restarts. A capture with a successful historical HTTP status and HTML MIME type supplies the metadata.

Example from the verified live dataset (no current-URL comparison supplied):

```json
{
  "url": "http://docs.python.org:80/3/",
  "domain": "docs.python.org",
  "capturedAt": "2012-10-29T03:46:10Z",
  "archiveUrl": "https://web.archive.org/web/20121029034610/http://docs.python.org:80/3/",
  "archivedStatusCode": 200,
  "mimeType": "text/html",
  "contentDigest": "6ILGSKJ2N36XB5EF7K2TMR75FTS7LRJO",
  "inCurrentUrls": null
}
```

| Field | Meaning |
| --- | --- |
| `url` | Original archived page URL. |
| `domain` | Hostname from the original URL. |
| `capturedAt` | UTC time of the selected archived capture, not publication time. |
| `archiveUrl` | Wayback link for this capture; replay availability can change. |
| `archivedStatusCode` | Historical index status; currently only successful 200 captures are returned. |
| `mimeType` | HTML or XHTML MIME type. |
| `contentDigest` | Archive-provided content fingerprint, or null if unavailable. |
| `inCurrentUrls` | True or false against your supplied list; null when no list is supplied. |

`inCurrentUrls: false` means the recovered URL is missing from the list you supplied. It does **not** establish that the current page is deleted, broken or returning 404.

HTTP failure captures, CDX headers, resume markers, invalid records and duplicates are excluded from the dataset. A stored HTTP 200 is index evidence: without downloading page bodies, the Actor cannot identify soft 404 pages or judge the contents of a URL such as `404.html`. The separate **Coverage and stop reason** output links to a run summary containing delivered counts, rejected records, source failures and whether coverage is partial.

### Coverage and recovery

The result limit can stop an export before the archive is exhausted. A successful run may therefore have partial coverage. Check the summary's `complete`, `partial`, `stopReason`, `sourceError` and `hasMoreResults` fields before treating an inventory as exhaustive.

If Internet Archive returns an access denial or remains unavailable after bounded retries, the Actor preserves delivered rows and reports the source problem. A source failure with no delivered rows fails the run instead of presenting an empty archive as success. A genuinely empty valid CDX response can complete with zero records.

Apify's resurrection feature can continue a stopped run with its original input and default storage. The Actor reconciles the dataset and drains saved pending rows before requesting another page. Starting a fresh run creates a separate inventory; it does not automatically reuse another run's checkpoint.

`hasMoreResults` is true when pending rows or a usable continuation remain, false only for verified complete coverage, and null when more results cannot be established. `recoveryAdvice` explains the appropriate next step. A result-limit stop needs a new run with a larger `maxResults`; resurrection keeps the original input limit. A spending-limit stop needs a budget review: resurrection does not automatically increase the original cap.

Graceful interruption writes the summary and URL-quality report. Hard platform termination can prevent final reports from being refreshed; after resurrection, delivered records are reconciled before continuing. Compare a report's analyzed count with the dataset count when inspecting a stopped run.

### Advisory URL quality

The **Advisory URL quality** output links to a separate, uncharged `URL_QUALITY` report. Dataset rows retain the same eight fields and their original URLs. The report contains `analyzedUrlCount`, `flaggedUrlCount`, per-flag `counts`, and `entries` with each flagged `url` and its `flags`.

- `tracking_query`: query keys beginning with `utm_`, or `_ga`, `_gl`, `gclid`, `fbclid` or `msclkid` (case insensitive).
- `error_like_path`: a final path name of `404`, `500`, `error`, `not-found` or `not_found`, ignoring its file extension.

These are advisory URL patterns. A legitimate research page named `404.html` may be flagged; the report cannot establish spam, soft 404s or page content. No flagged URL is automatically removed or rewritten. Rejected, pending and duplicate rows do not add warnings. Reports are reconstructed from delivered records after recovery and written at completion or graceful interruption, without a report write for every URL.

### Pricing

Pricing is **$0.0171 per 1,000 delivered URLs** ($0.0000171 per URL), plus **$0.0032 per run start until 23 October 2026** and **$0.0061 per run start from 23 October 2026 (00:00 UTC)** at the fixed 256 MB allocation. Platform usage is included. Before that change, a 100-URL run costs $0.00491 in events and a 1,000-URL run costs $0.0203. From 23 October, those totals are $0.00781 and $0.0232 respectively. The higher startup rate covers the measured cost of bounded upstream timeouts. See the Pricing tab for the effective configuration.

One validated unique dataset row triggers one URL event. Retries, rejected records, duplicates and status summaries are not URL events. The startup event applies even when the archive is empty or unavailable. A fresh run and each resurrection incur another startup event. Resurrection preserves the original dataset and URL-event counts; already delivered URLs are not emitted again.

See the Pricing tab for the current event rates before starting a run.

Use a **positive** Apify maximum spending limit to cap a run; zero is treated by the platform as no explicit limit. Owner tests can also consume platform usage within their account budget. When the event budget runs out, the Actor stops promptly and preserves pending rows. A result-count limit controls inventory size; it does not establish a spending limit across multiple runs.

### Python API example

Install the official `apify-client` package and provide your Apify token through your application's environment.

```python
import os
from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("thescrapelab/wayback-machine-url-recovery").call(
    run_input={"website": "https://docs.python.org/3/", "maxResults": 100},
    max_total_charge_usd=Decimal("0.25"),
)
if run is None:
    raise RuntimeError("Actor call returned no run or storage metadata")
if run is None or (run["status"] if isinstance(run, dict) else run.status) != "SUCCEEDED":
    raise RuntimeError("Check the Actor run log and coverage summary")

for item in client.dataset((run["defaultDatasetId"] if isinstance(run, dict) else run.default_dataset_id)).iterate_items():
    print(item["url"], item["archiveUrl"])

summary = client.key_value_store((run["defaultKeyValueStoreId"] if isinstance(run, dict) else run.default_key_value_store_id)).get_record("RUN_SUMMARY")
print(summary["value"] if summary is not None else "No summary available")
quality = client.key_value_store((run["defaultKeyValueStoreId"] if isinstance(run, dict) else run.default_key_value_store_id)).get_record("URL_QUALITY")
print(quality["value"] if quality is not None else "No quality report available")
```

This example uses Apify Python client 3.x. Run metadata uses typed objects; dataset rows and key-value records are dictionaries. Report contents are under `value`.

### Limitations and responsible use

- The Wayback Machine does not archive every website or every page. Missing index results do not prove that a URL never existed.
- Captures can be excluded, removed or unavailable for replay. The Actor does not bypass exclusions or restricted access.
- This product currently inventories successful HTML pages. Images, JavaScript, PDFs, failed captures and archived page contents are outside its output.
- A digest is archival metadata; it does not identify a price change, policy change or publication date.
- Internet Archive is an independent upstream service. Its availability and response times affect runs; each run makes sequential CDX requests at least three seconds apart, uses bounded retries, and honors Retry-After. Run multiple website jobs sequentially to avoid multiplying the source request rate.
- Use archive metadata responsibly and respect source restrictions. Do not collect or store personal data about anyone.
- Archived links do not grant rights to reuse the underlying content. Review [Internet Archive's terms](https://archive.org/about/terms) and applicable rights for your intended use.

The source is the [Internet Archive CDX API](https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md). This Actor is independently developed by TheScrapeLab and is not affiliated with or endorsed by Internet Archive.

### Support

Use the Actor's Issues tab and include the run ID, website scope and the summary's stop reason. Never include API tokens or private credentials.

# Actor input Schema

## `website` (type: `string`):

Enter a public domain (including subdomains) or a folder URL. Only successful archived HTML pages are returned. No login or API key needed.

## `maxResults` (type: `integer`):

Maximum unique URLs to deliver. Start small; increase when you need a larger inventory.

## `currentUrls` (type: `array`):

Paste your current URLs to identify archived URLs missing from this supplied list. This does not check whether a live page is deleted. Leave empty for URL recovery only.

## Actor input object example

```json
{
  "website": "https://www.iana.org/domains/reserved",
  "maxResults": 5,
  "currentUrls": []
}
```

# Actor output Schema

## `results` (type: `string`):

All unique delivered URL records in JSON, retaining the eight documented fields.

## `summary` (type: `string`):

Coverage, stop reason, recovery guidance and matched/missing counts for delivered URLs.

## `quality` (type: `string`):

Uncharged URL-pattern review for delivered records only; flags are advisory.

## `csv` (type: `string`):

Spreadsheet-friendly CSV of the same delivered dataset, with the original field names.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "website": "https://www.iana.org/domains/reserved",
    "maxResults": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("thescrapelab/wayback-machine-url-recovery").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "website": "https://www.iana.org/domains/reserved",
    "maxResults": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("thescrapelab/wayback-machine-url-recovery").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "website": "https://www.iana.org/domains/reserved",
  "maxResults": 5
}' |
apify call thescrapelab/wayback-machine-url-recovery --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thescrapelab/wayback-machine-url-recovery"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/E7U4ROzYAQ6Ovu2km/builds/HCveXg9jUaXCbcjQb/openapi.json