# Wayback Machine — Historical URLs and Snapshot History (`dalbian/wayback-machine-url-history`) Actor

Pull every URL the Internet Archive ever captured for a domain, with the dates, HTTP status and content type of each capture. Recover lost pages after a migration, audit a domain's past before buying it, or track when a single page changed.

- **URL**: https://apify.com/dalbian/wayback-machine-url-history.md
- **Developed by:** [Benjamin Jerez](https://apify.com/dalbian) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Machine — Historical URLs and Snapshot History

Ask the Internet Archive what it ever saw on a domain. Every URL it captured, when, what HTTP status the page returned, what type it was, how large, and whether the content had changed since the previous crawl.

Three questions, three modes: **what pages used to exist**, **how one page changed over time**, and **what this domain's past looks like** before you buy it.

### Unique URLs — the sitemap of what existed

The mode you want after a migration or a redesign. One row per distinct page the Archive ever captured, filtered to real pages rather than assets.

Filtering `apify.com` to `text/html`, status `200`, excluding query strings:

| First seen | Status | URL |
| --- | --- | --- |
| 2007-05-31 | 200 | http://www.apify.com:80/ |
| 2024-05-22 | 200 | https://apify.com/0code0 |
| 2024-05-29 | 200 | https://apify.com/0code0/yellowpages-in-categories/api/client/curl |

That list is where lost pages come from. A site that moved CMS, dropped a language, or retired a blog leaves hundreds of URLs that still hold links and still get requested. The Archive remembers them when the current sitemap does not.

The **URL must not contain** filter is what makes the output usable. Excluding `?replytocom`, `/feed/`, `/wp-json/` and similar removes the crawl noise that otherwise dominates a WordPress site.

### Snapshots — how a page changed

Every capture of one URL, or turn on **only captures where the content changed** and get just the dates it actually changed. That uses the Archive's own content checksum, so twenty identical weekly crawls collapse to one row.

`https://apify.com/store`, content changes only:

| Captured | Status | Bytes | Content hash |
| --- | --- | --- | --- |
| 2019-07-04T11:02:25Z | 200 | 72 893 | TTEN2TCX6TW4… |
| 2019-07-19T10:33:07Z | 200 | 54 607 | QW6RQLSJAIPU… |
| 2019-07-22T22:30:46Z | 200 | 54 696 | YNGGY3JVFB6B… |

The size column reads as a history on its own: that page lost eighteen kilobytes in a fortnight.

### Overview — a domain's past in one row

For anyone about to buy a domain, or trying to work out what a competitor used to be.

`dereso.fr`:

| Field | Value |
| --- | --- |
| `firstCapture` / `lastCapture` | 2016-06-23 · 2026-06-16 |
| `totalCaptures`, `uniqueUrls` | 2 865 · 996 |
| `yearsCovered` | 2016 through 2026, unbroken |
| `statusBreakdown` | 200: 2 009 · 301: 174 · 404: 20 · 405: 5 |
| `topContentTypes` | text/html 743 · image/png 376 · application/javascript 264 |

Eleven unbroken years with a healthy status mix is a domain that was a real site throughout. A gap of several years, a sudden explosion of thin pages, or a status mix dominated by redirects tells the opposite story — and that is exactly what a dropped-domain listing will not tell you.

### Who uses this

SEO and migration teams recovering URLs after a replatform, and finding pages worth redirecting rather than losing. Domain investors checking a name's history before bidding. Competitive researchers tracking when a rival changed pricing or messaging. Journalists and legal teams establishing what a page said on a given date. Anyone rebuilding a site whose CMS is gone.

### What it is and is not

This reads the Archive's **index**, not the pages. You get the URL, the date and the metadata of every capture, plus a `waybackUrl` that opens the archived page itself. It does not download page content.

**A capture is not proof the page was live.** The Archive records what it received, including 404s and redirects — which is useful, since a page already broken in 2019 is worth knowing about. The status filter is how you separate the two.

**`warc/revisit` is normal.** It marks a capture where the content was unchanged from a previous one, so the Archive stored a pointer rather than a copy. Those rows carry no status code of their own, which is why some rows show an empty status.

**Coverage is uneven.** The Archive crawls popular sites often and obscure ones rarely. An absent URL means it was never captured, not that it never existed.

### Source and licence

The **Internet Archive**'s public CDX index at `web.archive.org/cdx/search/cdx`. No key, no account. Queried live on every run.

One behaviour worth knowing, because it would otherwise look like a bug in this Actor: the CDX index intermittently answers a valid query with an empty response. The same request repeated three times returned results, nothing, then results again. Reporting that as "never archived" would be a serious error, so an empty answer is retried up to five times before it is believed.

### Frequently asked

**Why does a domain filter return so little?**
Because a scheme was included. `https://example.com` is read as a URL, and matching around a URL is narrower than matching around a domain. Bare domains are passed through as given, and a scheme is stripped automatically.

**How far back does it go?**
To 1996 in principle. In practice, to the first time anyone crawled that site — 2007 for `apify.com`, 2016 for `dereso.fr`.

**How many URLs can one run return?**
As many as you set. The index is paged with a resume key, so a large site is one run rather than many. Captures run into the hundreds of thousands on big domains, which is why the per-input cap exists.

**Can I get the page content?**
Not from here. Each row carries a `waybackUrl` pointing at the archived copy, which you can fetch or open.

**Why do two rows have the same URL in Snapshots mode?**
Because the page was captured twice. Turn on **only captures where the content changed** to keep just the crawls where something was different.

# Actor input Schema

## `urls` (type: `array`):

One per line. A bare domain such as example.com, or a full URL. What is matched around it is decided by the Match type below.

## `mode` (type: `string`):

Unique URLs is the sitemap of everything that ever existed. Snapshots is the history of individual pages. Overview is a quick read on a domain you are considering buying.

## `matchType` (type: `string`):

Use Exact when you are following the history of a single page.

## `dateFrom` (type: `string`):

Earliest capture to consider. Accepts a year (2019), a month (201903) or a day (20190315).

## `dateTo` (type: `string`):

Latest capture to consider, same formats.

## `statusCodes` (type: `array`):

Keep only captures that returned these codes, one per line — 200 for pages that worked, 301 for redirects, 404 for pages already broken when archived. Leave empty for all.

## `contentTypes` (type: `array`):

Keep only these MIME types, one per line, such as text/html or application/pdf. Filtering to text/html is what turns a raw capture list into a page list.

## `urlContains` (type: `array`):

Keep only URLs containing one of these strings, one per line. Combined with OR.

## `urlExcludes` (type: `array`):

Drop URLs containing any of these, one per line. Useful for stripping out ?replytocom, /wp-json/, /feed/ and other crawl noise.

## `onlyContentChanges` (type: `boolean`):

In Snapshots mode, collapse consecutive identical captures so you get the dates a page actually changed rather than every crawl. Uses the Archive's own content checksum.

## `maxResultsPerUrl` (type: `integer`):

Cap per domain or URL. A large site can hold hundreds of thousands of captures, so this is both a cost and a run-time control.

## Actor input object example

```json
{
  "urls": [
    "apify.com"
  ],
  "mode": "urls",
  "matchType": "domain",
  "contentTypes": [
    "text/html"
  ],
  "onlyContentChanges": false,
  "maxResultsPerUrl": 1000
}
```

# Actor output Schema

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "apify.com"
    ],
    "contentTypes": [
        "text/html"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("dalbian/wayback-machine-url-history").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["apify.com"],
    "contentTypes": ["text/html"],
}

# Run the Actor and wait for it to finish
run = client.actor("dalbian/wayback-machine-url-history").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "apify.com"
  ],
  "contentTypes": [
    "text/html"
  ]
}' |
apify call dalbian/wayback-machine-url-history --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dalbian/wayback-machine-url-history"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ekwO43jgyYBptTTNL/builds/X0cnGrwMHU3NtSueW/openapi.json