# Wayback URL Finder: Lost Pages, History & Redirect Gaps (`urlwright/wayback-url-finder`) Actor

Find archived URLs, domain history and relaunch redirect gaps from the Wayback Machine, then check which of them still resolve. Metadata only. Failed, blocked and duplicate URLs are free.

- **URL**: https://apify.com/urlwright/wayback-url-finder.md
- **Developed by:** [Urlwright](https://apify.com/urlwright) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 archived urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback URL Finder: Lost Pages, History & Redirect Gaps

**Urlwright.** Little tools for the web.

Old URLs of any domain, checked today for redirect gaps. The actor reads the public Wayback Machine and, when you ask it to, requests those URLs on the live web. You get metadata: the URL, when it was seen, the archived status, the MIME type, a content digest, and a link to the capture. Page bodies are never stored.

Uses the public Wayback Machine CDX API of the Internet Archive. Not affiliated with or endorsed by the Internet Archive.

### What it does

1. You pass domains (`example.com`) or URL prefixes (`https://example.com/blog/`). A dataset from Sitemap URL Extractor, Tech Stack Detector or Brand Kit can supply more targets.
2. **Lost URLs** lists each archived URL once, with the first and last time it was captured, how many snapshots were collapsed, the archived status, MIME type and digest, and links to the first and last capture.
3. **Domain history** returns one row per target: first and last capture, captures per year, years with no captures, and a sample of status and MIME types.
4. **URL lookup** returns the capture closest to a date you choose.
5. Optional **live check**: HEAD, then GET if the server rejects HEAD. Redirects are followed (at most five). `robots.txt` for UrlwrightBot is respected unless you turn that off.
6. Optional **sitemap comparison**. Pass a Sitemap URL Extractor dataset and each URL is marked `inSitemap`. A redirect gap is an archived HTML page that today returns 404 or 410, was not redirected, and (when you passed a sitemap) is not in that sitemap.
7. URLs that contain an email address, or a session, token or password parameter, are dropped and counted. They are not written out.

One target that the archive cannot answer does not fail the run. `RUN_SUMMARY` is always written to the key value store, including when the input is rejected.

### Who it is for

- Relaunches: find URLs the new site forgot, so backlinks do not end on a 404.
- Buying a domain: see the years it was captured, and the years it was not.
- Showing when a page was public: the row carries archive links. Evidential value must be checked by a lawyer.

### Input

The console prefills a small run (`https://urlwright.com`, 25 URLs, 10 live checks). Defaults for a real run are higher: 1,000 URLs per target and 200 live checks.

```json
{
  "startUrls": ["https://urlwright.com", "https://example.com/blog/"],
  "mode": "lostUrls",
  "includeSubdomains": false,
  "from": "2018",
  "to": "2024",
  "mimeTypes": ["text/html"],
  "statusCodes": ["200"],
  "maxUrlsPerTarget": 1000,
  "checkLiveStatus": true,
  "maxLiveChecksPerTarget": 200,
  "respectRobotsTxt": true,
  "sitemapDatasetId": ""
}
```

`datasetId` plus `datasetField` (default `url`) reads extra targets. If the named field is missing on a row, the actor tries `url`, then `finalUrl`, then `domain`, then `inputUrl`.

`includeSubdomains` false asks for a domain once, with `matchType=domain` filtered to the apex host and `www`. True queries the host and its subdomains. A path is still queried on the host and its www twin, because a domain query is not a path prefix. A host that already has a subdomain label, such as `blog.example.com`, is queried as written.

Dates are `YYYY`, `YYYY-MM` or `YYYY-MM-DD`. `from` must be on or before `to`. `lookupDate` is required when `mode` is `lookup`.

### Output

One dataset, two row types. Every field is present. Unused fields are null.

An archived URL:

```json
{
  "schemaVersion": 1,
  "type": "archived-url",
  "inputUrl": "https://example.com",
  "domain": "example.com",
  "url": "http://example.com/about",
  "status": "ok",
  "httpStatus": 404,
  "finalUrl": "http://example.com/about",
  "redirectChain": [],
  "archiveHttpStatus": 200,
  "mime": "text/html",
  "digest": "ABCDEF",
  "firstSeen": "2016-03-01T00:00:00Z",
  "lastSeen": "2019-11-02T00:00:00Z",
  "snapshotCount": 4,
  "firstArchiveUrl": "https://web.archive.org/web/20160301000000/http://example.com/about",
  "lastArchiveUrl": "https://web.archive.org/web/20191102000000/http://example.com/about",
  "inSitemap": false,
  "redirectGap": true,
  "charged": true,
  "chargedEvents": { "archived-url": 1, "domain-history": 0, "live-status-checked": 1 },
  "error": null
}
```

`mime` is the MIME type of the archived capture. `httpStatus` is the status today. `snapshotCount` is null when the archive does not return a group count. Views: Overview, Redirect gaps, History.

Live checks are the only requests sent to a customer host. CDX requests go only to web.archive.org. When the SSRF guard blocks a live check, on the first request or on any redirect, `error` is `ssrf-blocked`, `httpStatus` is null and `redirectGap` is null. That live check is not billed. The `archived-url` charge on the same row still applies, because the archive metadata was delivered.

`RUN_SUMMARY` counts rows, malformed archive lines, filtered personal URLs, and `stoppedReason` (`charge-limit` or `timeout`) when the run stops early. `REDIRECT_GAPS.csv` is written when at least one gap was found.

A domain history row uses `type: "domain-history"` and `status` `ok`, `partial`, `no-captures`, `archive-unavailable` or `invalid`. `distributionSample` is true: the status and MIME counts come from collapsed archive rows, not from every capture. `partial` is true, and `status` is `partial`, when a query was capped or failed. That row is free. Only `status: ok` is billed.

### Pricing

Pay per event. No start fee. A failed archive query, a duplicate row and a filtered personal URL are free. A live check blocked by the SSRF guard is reported as blocked and is not billed. The archived URL on that row is still billed, because the archive metadata was delivered. Prices in USD. PLATINUM and DIAMOND use the GOLD price. The minimum max charge per run is $0.02.

| Event | What you pay for | FREE | BRONZE | SILVER | GOLD |
|---|---|---:|---:|---:|---:|
| `archived-url` | One archived URL written to the dataset | 0.0005 | 0.00045 | 0.0004 | 0.0003 |
| `domain-history` | One domain history with status ok | 0.006 | 0.0055 | 0.0052 | 0.005 |
| `live-status-checked` | One live check that received an HTTP status | 0.0005 | 0.00045 | 0.0004 | 0.0003 |

A failed or unavailable archive query is never billed. Small runs usually finish in 30 to 120 seconds.

At the FREE price, 1,000 archived URLs are $0.50. The same 1,000 rows with a live status are $1.00. A domain history is $0.006.

### Limits and runtime

The Internet Archive asks callers to slow down. This actor sends at most 0.8 CDX requests per second, one at a time, and does not rotate IPs. A page asks for at most 2,000 collapsed URLs, and only as many as the run can still keep, plus a small buffer. 1,000 URLs are usually one request: a few seconds when the archive is quick, plus 1.25 seconds between requests. Larger lists add at least 1.25 seconds per extra page. Saved rows are written and billed in batches, so a long list does not wait on a separate platform call for every URL. Each CDX attempt waits at most 30 seconds, and one target stops after three attempts or 120 seconds of archive time, whichever comes first. Live checks add about one second per request on a single host, with up to four hosts at once.

The archive can be down or slow. A target then comes back as `archive-unavailable`, with the last HTTP status or timeout in `error`, and is not billed. A history that is only partly filled comes back as `partial` and is not billed. Personal URLs are removed before you see them.

Suggested run options: build `latest`, 128 MB memory, timeout 3600 seconds. Memory can be raised up to 1024 MB. Measured runs peaked at 31 to 45 MB.

### FAQ

**Does it download old pages?** No. Metadata and archive links only.

**Can I use this as proof in a dispute?** The archive links show what the Wayback Machine stored. Evidential value must be checked by a lawyer.

**Why are some URLs missing?** The archive may not have captured them, the MIME type or status filter excluded them, or a personal parameter caused them to be dropped. `RUN_SUMMARY` reports the counts.

**What is a redirect gap?** Archived HTML that returned a 2xx status, and today returns 404 or 410 without a redirect. If you passed a sitemap dataset, the URL also has to be absent from that sitemap.

**Will a failed target fail the run?** No. Other targets continue. The summary explains each one.

### No warranty

The output depends on the Internet Archive and on the sites you check. Captures can be missing, incomplete or wrong. Nothing in the dataset is legal advice, and no warranty is given that a URL was public, or that a live status will stay the same.

# Actor input Schema

## `startUrls` (type: `array`):

Domains (example.com) or URL prefixes (https://example.com/blog/). Scheme, port and query are removed for domains. At least one value is required unless datasetId is set.

## `datasetId` (type: `string`):

Read more targets from an Apify dataset, for example the output of Sitemap URL Extractor, Tech Stack Detector or Brand Kit.

## `datasetField` (type: `string`):

Field that holds the domain or URL. When that field is missing on a row, url, finalUrl, domain and inputUrl are tried, in that order.

## `sitemapDatasetId` (type: `string`):

Dataset from Sitemap URL Extractor. Rows are read from the url field and matched to archived URLs (http and https, www and a trailing slash count as the same URL). Used for redirect gaps.

## `mode` (type: `string`):

lostUrls lists archived URLs. history returns one domain history row per target. lookup returns the capture closest to lookupDate.

## `lookupDate` (type: `string`):

Required for lookup mode. YYYY, YYYY-MM or YYYY-MM-DD. The closest archived capture to the start of that period is returned.

## `includeSubdomains` (type: `boolean`):

When true, the archive query uses matchType domain (the host and its subdomains). When false, a domain is one matchType domain query filtered to the apex host and www. A path is still queried on the host and its www twin. Hosts with more than two labels are queried as written.

## `from` (type: `string`):

Earliest capture to include. YYYY, YYYY-MM or YYYY-MM-DD. Must be on or before To.

## `to` (type: `string`):

Latest capture to include. YYYY, YYYY-MM or YYYY-MM-DD.

## `mimeTypes` (type: `array`):

Archived MIME types to keep in lost URL and lookup modes. Empty means all types. History mode reports the distribution and does not apply this filter.

## `statusCodes` (type: `array`):

Archived HTTP status codes to keep. Use a code such as 200 or a class such as 2xx, 3xx, 4xx or 5xx. Empty means all codes.

## `maxUrlsPerTarget` (type: `integer`):

Stop listing archived URLs for a target after this many rows. Further captures are not returned and the run is marked truncated.

## `maxTargets` (type: `integer`):

Maximum number of targets after normalisation and deduplication. The run rejects the input when there are more.

## `checkLiveStatus` (type: `boolean`):

Request each archived URL today (HEAD, then GET if the server rejects HEAD) and fill httpStatus, finalUrl and redirectGap. Respects robots.txt unless you turn that off.

## `maxLiveChecksPerTarget` (type: `integer`):

How many archived URLs per target are requested on the live web. The rest keep a null httpStatus.

## `respectRobotsTxt` (type: `boolean`):

Read robots.txt for UrlwrightBot before each live check. A disallow, or a robots.txt that cannot be fetched, skips the check and is not billed.

## Actor input object example

```json
{
  "startUrls": [
    "https://urlwright.com"
  ],
  "datasetField": "url",
  "mode": "lostUrls",
  "lookupDate": "2020-01-01",
  "includeSubdomains": false,
  "mimeTypes": [
    "text/html"
  ],
  "statusCodes": [
    "200"
  ],
  "maxUrlsPerTarget": 25,
  "maxTargets": 5,
  "checkLiveStatus": true,
  "maxLiveChecksPerTarget": 10,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `overview` (type: `string`):

Dataset rows. type is archived-url or domain-history. Missing values are null.

## `redirectGaps` (type: `string`):

Archived HTML URLs that now return 404 or 410 without a redirect.

## `history` (type: `string`):

One history row per target in history mode, or a status row when a target produced no URLs.

## `runSummary` (type: `string`):

Counts, charged events, malformed CDX rows, filtered personal URLs and why the run stopped.

## `redirectGapsCsv` (type: `string`):

CSV of redirect gaps, written only when at least one gap was found.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://urlwright.com"
    ],
    "lookupDate": "2020-01-01",
    "mimeTypes": [
        "text/html"
    ],
    "statusCodes": [
        "200"
    ],
    "maxUrlsPerTarget": 25,
    "maxTargets": 5,
    "maxLiveChecksPerTarget": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("urlwright/wayback-url-finder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://urlwright.com"],
    "lookupDate": "2020-01-01",
    "mimeTypes": ["text/html"],
    "statusCodes": ["200"],
    "maxUrlsPerTarget": 25,
    "maxTargets": 5,
    "maxLiveChecksPerTarget": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("urlwright/wayback-url-finder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://urlwright.com"
  ],
  "lookupDate": "2020-01-01",
  "mimeTypes": [
    "text/html"
  ],
  "statusCodes": [
    "200"
  ],
  "maxUrlsPerTarget": 25,
  "maxTargets": 5,
  "maxLiveChecksPerTarget": 10
}' |
apify call urlwright/wayback-url-finder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,urlwright/wayback-url-finder"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/GScpNY64b3phaHZHV/builds/EWEZ6PacsE8gRGvG5/openapi.json