# Wayback Lost Pages Finder (`sculptural_delphinium/wayback-lost-pages`) Actor

List every page of a domain the Wayback Machine archived, then check each one on the live site today: still live, redirected, redirected to the homepage, or lost (404/410). Use it to recover traffic after a migration.

- **URL**: https://apify.com/sculptural_delphinium/wayback-lost-pages.md
- **Developed by:** [Josep Peruga](https://apify.com/sculptural_delphinium) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Lost Pages Finder

Find the pages your site used to have that are gone today. The Actor asks the Internet Archive's Wayback Machine for every HTML page it ever archived on a domain, then checks each one on the live site and tells you what happens to it now:

| Status | Meaning |
|---|---|
| `lost` | The old URL returns 404 or 410. Backlinks and bookmarks to it are dead |
| `redirected-to-homepage` | The old URL redirects to the homepage. Google usually treats this as a soft 404 |
| `redirected` | The old URL redirects to another page that loads |
| `live` | The page still loads at the same address (an http → https or www switch counts as the same address) |
| `error` | 5xx, other 4xx, timeout, redirect loop or too many hops |
| `not-checked` | robots.txt disallows the URL, so it was not requested (not charged) |

Use it after a migration, a redesign, a CMS change or a domain move to find the URLs nobody put in the redirect map, or before you buy or relaunch a domain to see what it used to contain.

### Input

- **Domains**: one or more domains, e.g. `example.com`. `www.` is folded into the bare domain.
- **Max pages per domain** (default 500): the cap on unique archived pages checked.
- **Include subdomains**: also list `blog.example.com`, `shop.example.com` and so on.
- **Archived from / up to year**: restrict to captures from a period, e.g. up to the year before your migration.
- **Include URLs with query strings**: off by default, because archived `?utm=` and session variants are usually duplicates.
- **Skip file extensions**: the defaults drop assets (css, js, images, fonts, pdf, office files, media).

### Output

One dataset row per archived page:

```json
{
  "domain": "crawlee.dev",
  "archivedUrl": "https://crawlee.dev/api/3.0/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "firstArchived": "2023-06-06T07:10:04Z",
  "firstSnapshotUrl": "https://web.archive.org/web/20230606071004/https://crawlee.dev/api/3.0/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "latestSnapshotUrl": "https://web.archive.org/web/https://crawlee.dev/api/3.0/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "status": "lost",
  "finalUrl": "https://crawlee.dev/js/api/browser-crawler/interface/BrowserCrawlerEnqueueLinksOptions",
  "finalStatus": 404,
  "hops": 2,
  "note": null,
  "checked": true,
  "checkedAt": "2026-10-28T09:50:14.863Z"
}
```

`finalUrl` is where the redirect chain ends, so a `lost` row with `hops > 0` is a redirect that points at a missing page.

The key-value store also has:

- **LOST_URLS**: the `lost` URLs as plain text, one per line. Add a target to each and check the result with our [Redirect Map Validator](https://apify.com/sculptural_delphinium/redirect-map-validator).
- **OUTPUT**: per-domain counts for each status, and whether the archive list was complete or hit your page cap.

### How it works

1. It queries the Wayback Machine CDX API (the Internet Archive's public index) for captures that returned HTTP 200 with type `text/html`, one entry per unique URL. Requests are paced at about one per 1.5 seconds, with back-off on errors.
2. Duplicates are merged: http/https, www/non-www and a trailing slash count as one page.
3. Each page is requested on the live site. Redirects are followed one hop at a time (up to 10). No page bodies are downloaded, only status codes and headers.

### How it behaves on your site

- It identifies itself as `WaybackLostPages/1.0` and obeys robots.txt for that agent (or `*`), including `Crawl-delay` up to 10 s per host. Turn **Respect robots.txt** off only for sites you own or are authorised to test.
- It collects no personal data. It records URLs, status codes and capture dates.

### Pricing

Pay per event: each archived page checked on the live site is one `page-checked` event. Pages skipped because of robots.txt are not charged. Use **Max pages per domain** and the run's maximum cost setting to cap spend. The run stops cleanly when the cap is reached.

### Limitations

- The Wayback Machine only knows pages it crawled. Small or new sites may have few captures, and pages blocked from the archive are missing.
- Big domains have hundreds of thousands of archived URLs. The list is read in archive order (by URL path) until your cap is reached, so use a year range or a higher cap for full coverage.
- A page that returns 200 with "not found" content (a soft 404) shows as `live`.
- If the Wayback Machine is down or rate-limits the run, that domain is reported with an error in OUTPUT and nothing is charged for it.

### Support

Open an issue on the Actor's Issues tab. Breakages are fixed within 48 hours.

*This Actor is built and maintained by an AI agent working for its publisher. A human owns the account and is responsible for it. It is not affiliated with the Internet Archive.*

# Actor input Schema

## `domains` (type: `array`):

Domains to look up, for example example.com. A full URL also works; only its host is used, and www. is folded into the bare domain.

## `maxUrlsPerDomain` (type: `integer`):

Upper limit on unique archived pages checked per domain. Each page checked on the live site is one billable event.

## `includeSubdomains` (type: `boolean`):

Also list pages archived on subdomains (blog.example.com, shop.example.com). Off: only example.com and www.example.com.

## `fromYear` (type: `integer`):

Optional. Only pages captured in this year or later.

## `toYear` (type: `integer`):

Optional. Only pages captured in this year or earlier. Useful to see what the site looked like before a migration.

## `includeQueryUrls` (type: `boolean`):

Off by default because archived ?utm=, ?sessionid= and tracking variants are usually duplicates of the same page.

## `skipExtensions` (type: `array`):

Archived URLs ending in these extensions are ignored. Only captures served as text/html are listed in the first place; this also drops misconfigured assets. Leave empty to use the default list (css, js, images, fonts, pdf, office files, archives, media).

## `respectRobotsTxt` (type: `boolean`):

Skip live pages that the site's robots.txt disallows, and honour Crawl-delay (up to 10 s). Skipped pages are listed as not-checked and are not charged. Turn off only for sites you own or are authorised to test.

## `maxConcurrency` (type: `integer`):

Parallel requests to the live site.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each request to the live site.

## Actor input object example

```json
{
  "domains": [
    "crawlee.dev"
  ],
  "maxUrlsPerDomain": 500,
  "includeSubdomains": false,
  "includeQueryUrls": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 4,
  "requestTimeoutSecs": 15
}
```

# Actor output Schema

## `results` (type: `string`):

One row per archived page: its status on the live site today, where it lands, HTTP status, first capture date and links to the archived copies.

## `lostUrls` (type: `string`):

The archived URLs that now return 404 or 410, one per line, ready to paste into a redirect map.

## `summary` (type: `string`):

Per-domain counts of live, redirected, redirected-to-homepage, lost and error pages, stored in the OUTPUT record.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("sculptural_delphinium/wayback-lost-pages").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("sculptural_delphinium/wayback-lost-pages").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "crawlee.dev"
  ]
}' |
apify call sculptural_delphinium/wayback-lost-pages --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sculptural_delphinium/wayback-lost-pages"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/t0H0bur5btaLvnpn8/builds/58OkXbgvCUXKcegaJ/openapi.json