# Wayback Machine URL History & Removed Pages Finder (`thomson-digital/wayback-url-history`) Actor

List every URL the Wayback Machine archived for a domain, with first/last seen dates and archive links. Find removed pages (404, gone, redirected to home) for SEO link reclamation, redirect maps and content recovery. Fast, no browser.

- **URL**: https://apify.com/thomson-digital/wayback-url-history.md
- **Developed by:** [Matt](https://apify.com/thomson-digital) (community)
- **Categories:** SEO tools, Developer tools, Lead generation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Machine URL History & Removed Pages Finder

List **every page the Internet Archive's Wayback Machine has captured for a website**, with its **first and last seen dates**, number of months captured, and direct links to the archived copies. Then let the Actor **check each URL on the live site** and flag pages that have been **removed** (404, 410, unreachable, or quietly redirected to the homepage).

Just enter a domain. No browser, no proxies, no login — it uses the public Wayback CDX index.

### What can it do?

- 🗂️ **Full URL inventory from the archive** for a whole domain (incl. subdomains), a single host, a path prefix like `/blog/`, or one exact URL
- 📅 **History per URL**: `firstSeen`, `lastSeen`, `captureMonths`, last recorded HTTP status, links to the first and last archived copies
- 🪦 **Removed pages finder**: checks each archived URL on the live site and returns `liveStatus` (`live`, `redirected`, `redirected-to-home`, `not-found`, `gone`, `unreachable` …) and `isRemoved`
- 📸 **Snapshot mode**: every capture of a page, collapsed per day / month / year or only when the content changed
- 🎯 Filters: date range, HTTP status, content type (HTML by default, add PDFs etc.), regex include/exclude
- 🛡️ Built for the archive's quirks: automatic retries with backoff on rate limits (429) and "temporarily offline" outages, paging with resume keys, per-target error report

### Use cases

- **SEO link reclamation** — find old URLs that still have backlinks but now return 404, and redirect them
- **Site migrations & redirect maps** — get every URL a site ever had, not only what's in today's sitemap
- **Content recovery** — find and restore lost articles, product pages or PDFs from archived copies
- **Expired / aged domain research** — see what a domain hosted and when, before you buy it
- **Competitor research** — see which pages, products or landing pages a competitor launched and retired over time
- **OSINT / security** — discover forgotten endpoints and files
- **AI agents & data pipelines** — simple JSON input/output via API or MCP

### Input example

```json
{
  "startUrls": ["example.com"],
  "scope": "domain",
  "mode": "urls",
  "maxResultsPerTarget": 100,
  "onlyRemovedPages": true
}
```

| Field | Description |
| --- | --- |
| `startUrls` | Domains or URLs. |
| `scope` | `domain` (incl. subdomains), `host`, `prefix` (path), `exact`. |
| `mode` | `urls` = one row per unique page with history; `snapshots` = one row per capture. |
| `maxResultsPerTarget` | Max rows per entry (default 100). |
| `fromDate` / `toDate` | Capture date range. |
| `statusFilter` | `200` (default), `2xx3xx`, `any`. |
| `mimeTypes` | Default `["text/html"]`. Empty = all files. |
| `includeUrlPatterns` / `excludeUrlPatterns` | Regex filters. |
| `checkLiveStatus` | Add live status to every row. |
| `onlyRemovedPages` | Output only pages that no longer exist. |
| `snapshotGranularity` | `all`, `day`, `month`, `year`, `content-change` (snapshots mode). |
| Advanced | `maxCdxPages`, `liveCheckConcurrency`, `requestTimeoutSecs`, `maxRetries`, `retryBudgetSecs` (caps time spent waiting on an unavailable archive). |

### Output example (urls mode with live check)

```json
{
  "target": "apify.com",
  "url": "https://apify.com/",
  "urlkey": "com,apify)/",
  "firstSeen": "2007-05-31T10:15:38Z",
  "lastSeen": "2026-10-01T04:51:28Z",
  "captureMonths": 125,
  "lastStatusCode": "200",
  "mimeType": "text/html",
  "firstArchiveUrl": "https://web.archive.org/web/20070531101538/http://www.apify.com:80/",
  "lastArchiveUrl": "https://web.archive.org/web/20261001045128/https://apify.com/",
  "liveStatusCode": 200,
  "liveFinalUrl": "https://apify.com/",
  "liveStatus": "live",
  "isRemoved": false
}
```

Snapshot mode rows contain `timestamp`, `capturedAt`, `archiveUrl`, `statusCode`, `mimeType`, `digest` and `length`. A per-target report is saved in the key-value store record `SUMMARY`.

### Pricing

Pay per event:

- **Result row** written to the dataset.
- **Live check** — only when `checkLiveStatus` or `onlyRemovedPages` is on, per URL checked on the live site. In "only removed pages" mode, all checked URLs are charged as live checks, but only removed pages are charged as results.

Nothing is charged when the archive has no captures or is unavailable. Set a maximum cost per run and the Actor stops cleanly at the limit.

### FAQ

**Why do some runs take a while?** The Wayback CDX API can be slow for large domains and occasionally rate-limits or goes offline. The Actor retries automatically (up to `maxRetries`) and reports any problem in `SUMMARY`.

**What does `captureMonths` mean?** The number of distinct months in which the archive captured that URL — a good proxy for how long and how prominently a page existed.

**How is "removed" decided?** `isRemoved` is true when the live URL returns 404 or 410, the host is unreachable, or the page now redirects to the site's homepage (a common "soft 404"). Normal redirects to another page are reported as `redirected` and not counted as removed.

**Does it download page content?** No — it returns URLs, dates, metadata and archive links. Open `lastArchiveUrl` to view the archived copy.

**Is this allowed?** The Actor reads the Internet Archive's public index and makes one lightweight request per URL to the live site when you enable live checks. It collects no personal data. Please use the data responsibly and respect the Internet Archive's terms.

**Found a bug or need a feature?** Open an issue on the Issues tab.

# Actor input Schema

## `startUrls` (type: `array`):

One per line, e.g. example.com, blog.example.com or https://example.com/blog/. How each entry is matched is set by 'Scope'.

## `scope` (type: `string`):

What to look up in the Wayback Machine for each entry.

## `mode` (type: `string`):

'Unique URLs' gives one row per archived page with its history. 'Every snapshot' gives one row per capture (use with granularity).

## `maxResultsPerTarget` (type: `integer`):

Maximum rows output for each entry. Keep small for testing.

## `fromDate` (type: `string`):

Optional. Only captures on or after this date (YYYY-MM-DD).

## `toDate` (type: `string`):

Optional. Only captures on or before this date (YYYY-MM-DD).

## `statusFilter` (type: `string`):

Which captures to consider, by the HTTP status the archive recorded.

## `mimeTypes` (type: `array`):

Only captures with these MIME types. Default text/html (web pages). Add application/pdf for PDFs. Empty list = everything (images, CSS, JS...).

## `includeUrlPatterns` (type: `array`):

Optional, case-insensitive regular expressions, e.g. /blog/.

## `excludeUrlPatterns` (type: `array`):

Optional, e.g. ?utm\_ or /wp-json/.

## `checkLiveStatus` (type: `boolean`):

Requests every archived URL on the live site and adds liveStatus (live, redirected, redirected-to-home, not-found, gone, unreachable...) and isRemoved. Charged as an extra 'live check' event.

## `onlyRemovedPages` (type: `boolean`):

Find pages that existed in the past but are gone now (404/410, unreachable, or redirected to the homepage) - great for SEO link reclamation, redirect maps and content recovery. Implies the live check.

## `snapshotGranularity` (type: `string`):

In 'Every snapshot' mode, collapse captures of the same URL to reduce noise and cost.

## `maxCdxPages` (type: `integer`):

Safety cap on Wayback CDX API requests per entry (each returns up to 5,000 index rows).

## `liveCheckConcurrency` (type: `integer`):

Parallel requests to the live website when checking status.

## `requestTimeoutSecs` (type: `integer`):

The archive index can be slow for big domains.

## `maxRetries` (type: `integer`):

Retries when the Internet Archive rate-limits (429) or is temporarily offline, with exponential backoff.

## `retryBudgetSecs` (type: `integer`):

Upper bound on how long the Actor keeps retrying one archive index request (rate limits / outages) before giving up on that entry. Keeps runs short when the archive is down.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com"
  ],
  "scope": "domain",
  "mode": "urls",
  "maxResultsPerTarget": 20,
  "statusFilter": "200",
  "mimeTypes": [
    "text/html"
  ],
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "checkLiveStatus": false,
  "onlyRemovedPages": false,
  "snapshotGranularity": "month",
  "maxCdxPages": 20,
  "liveCheckConcurrency": 10,
  "requestTimeoutSecs": 90,
  "maxRetries": 5,
  "retryBudgetSecs": 210
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com"
    ],
    "maxResultsPerTarget": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("thomson-digital/wayback-url-history").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["apify.com"],
    "maxResultsPerTarget": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("thomson-digital/wayback-url-history").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com"
  ],
  "maxResultsPerTarget": 20
}' |
apify call thomson-digital/wayback-url-history --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thomson-digital/wayback-url-history"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zMVqhKQUUgh4lvbjD/builds/vwqYOlqHlff5loSne/openapi.json