# Wayback Machine Scraper - Snapshots & Every Archived URL (`neverempty/wayback-machine-scraper`) Actor

For site migrations, redirect maps, recovering deleted pages and domain diligence: every snapshot of a URL, or every archived URL under a domain, from the official Wayback CDX API - first and last capture, capture count, status codes and a link to the copy. One page alone can return 768,693 rows.

- **URL**: https://apify.com/neverempty/wayback-machine-scraper.md
- **Developed by:** [NeverEmpty](https://apify.com/neverempty) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.11 / 1,000 result returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wayback Machine Scraper - Snapshots & Every Archived URL

**Useful for recovering deleted pages, building redirect maps after a migration, auditing old subdomains, and checking what an expired domain used to host**: look up what the Internet Archive's Wayback Machine holds for any page or domain, straight from its official CDX API.

It does two things:

- **Snapshots** — one row per archived capture of a URL: when it was captured, the HTTP status the page returned, the content type and size, whether the content changed since the capture before, and a link to the archived copy (plus a link to the raw archived file with no Wayback toolbar).
- **Every archived URL** — one row per distinct URL the archive holds under a domain, host or path: when it was first and last captured, how many captures there are, every status code it has returned, and the latest status. In this mode every row already carries `statusCodesSeen`, so pages that ever returned a 404 can be picked straight out of the list.

**The archive's own API falls over on exactly the domains you care about, and all of it was measured on 2026-09-10 and 2026-09-11.** Asking for every unique URL under a domain the archive's own way **can time out with HTTP 504 after 60 seconds, even for 20 URLs (2026-09-10)** — so captures are read here in light **5,000**-row pages and grouped locally, which is why large domains work. With no limit, **one page alone can return 768,693 rows**, so every request carries a limit. And the archive's own paging silently skips captures at the page boundary: at apify.com/ on 2019-07-11 22:42:57 there are two captures, a 200 and a 301, and paging one at a time returns only the 200.

**Nothing is dressed up as data it is not.** About one capture in eight is a "revisit" with status `-` (**377 of 3,000** captures under apify.com): `statusCode` comes back `null` with `captureType: "revisit"`, never a fake `0`. 503 and 504 are routine, so each request is retried with increasing waits — and if it still fails, a row says so instead of reporting "no captures". google.com answered **504, 504, 504, 503, 504 over 285 seconds** on 2026-09-11, and that is reported as what it was. nytimes.com refuses URL listings outright with HTTP 403 whatever the date range, and that is reported too.

**The status filter is sent to the archive rather than applied afterwards**, because filtering after thinning out drops whole months — **5 of apify.com's latest 24** monthly captures are 301 redirects.

No API key, no browser, no proxy. Export as JSON, CSV or Excel.

### What the archive gets wrong, and what this Actor does about it

These were measured against the live CDX API on 2026-09-10.

| The archive's behaviour | What you get here |
|---|---|
| A mistyped domain and a broken date both come back as the same empty answer as a real page that was never archived | Entries are checked before they are sent. A malformed URL or impossible date is returned as its own row explaining why, never as "no captures" |
| A malformed URL such as `http://[bad` is silently answered with captures of a *different* address (`bad`) | Host names are validated; entries that are not a real domain or IPv4 address are not sent |
| About one capture in eight is a "revisit" with status `-` (377 of 3,000 captures under apify.com) | `statusCode` is `null` and `captureType` is `revisit` - never a fake `0` |
| Content types include `unk` and `unknown`, which are not content types | `mimeType` is `null` for those |
| Asking for every unique URL under a domain the archive's own way can time out (HTTP 504 after 60 seconds, even for 20 URLs on 2026-09-10) | Captures are read in light 5,000-row pages and grouped here, so large domains work |
| With no limit, one page alone can return 768,693 rows | Every request carries a limit |
| 503 "Temporarily Offline" and 504 timeouts are routine | Each request is retried with increasing waits; if it still fails, a row says so instead of reporting "no captures" |
| The most heavily archived pages overwhelm the archive's own servers. For google.com, asking for one capture per year timed out (HTTP 504 after 60 seconds) oldest-first and newest-first, and even single-year lookups failed 4 times in 7 | Each request is retried with increasing waits for up to about eight minutes (google.com took 285 seconds on 2026-09-11), and for less when the run is close to its time limit. If the archive still cannot answer, you get a `failed` row saying so - nothing is guessed. Pages with a normal amount of history (apify.com, python.org, docs.python.org, github.com, openai.com) answered in 3 to 25 seconds in the same tests |
| A run has a time limit (one hour by default), and those heavily archived pages take about five minutes each: google.com answered HTTP 504, 504, 504, 503, 504 over 285 seconds on 2026-09-11. Many of them in one run would reach the limit and the platform would cut the run off, leaving the remaining entries with no row at all | The Actor knows when the run will end. Each request's wait and the retries are shortened so that no lookup runs past that point, and when less than 75 seconds are left it stops querying and every remaining entry gets its own `skipped-time-limit` row. An entry whose retries were cut short this way gets the same status, and a URL list cut short part-way ends with an `incomplete` row |
| Some sites are refused for URL listings: nytimes.com answers every domain, host and path listing with HTTP 403 "This type of CDX query requires authorization", whatever the date range (apify.com, python.org and wikipedia.org list normally) | A `refused-by-archive` row says the archive declined, that retrying will not help, and that single-page snapshots of that site still work - it is not passed off as a temporary error or as "no captures" |
| The archive's own paging (its resume key) silently skips captures at the page boundary: at apify.com/ on 2019-07-11 22:42:57 there are two captures, a 200 and a 301, and paging one at a time returns only the 200 | Snapshots are read in one request (up to 10,000 rows), so nothing is paged. URL lists re-ask for the boundary second of every page and add what the archive skipped, without counting anything twice |
| Filtering by status after thinning out drops whole months: 5 of apify.com's latest 24 monthly captures are 301 redirects | The status filter is sent to the archive, which applies it before thinning out - the same 24 months all come back with a 200 |

If a URL list stops early (a page of the archive could not be read, the reading limit was reached, or the run was close to its time limit), every URL except the last one is still fully counted - the archive returns captures grouped by URL in date order - and that last one is marked `complete: false`, with a row explaining that the list is incomplete. A URL whose page-boundary re-check failed is also marked `complete: false`.

Addresses with a non-standard port (such as `:8080`) are rejected rather than looked up, because dropping the port would return captures of a different address.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` | - | Pages or domains. In URL-list mode, `*.example.com` means every subdomain and `example.com/blog/*` means everything under that path |
| `mode` | `snapshots` | `snapshots` or `urls` |
| `matchType` | `domain` | URL list only: `domain` (host and subdomains), `host`, or `prefix` |
| `from` / `to` | - | `YYYY`, `YYYY-MM`, `YYYY-MM-DD` or the archive's own digits (`20210315`). Inclusive |
| `collapse` | `month` | Snapshots only: one per `month`, `day` or `year`, only when the content `change`d, or `none` |
| `order` | `newest` | Snapshots only: `newest` or `oldest` first |
| `statusFilter` | `any` | Snapshots only: `ok` (2xx), `redirect` (3xx) or `error` (4xx/5xx). The archive applies it before thinning out, so with `collapse: month` you get one matching capture per month rather than losing the months whose first capture was a redirect. Revisits carry no status and only appear with `any` |
| `maxResultsPerUrl` | `100` | Up to 10,000 rows per entry |

If `urls` is empty, `apify.com` is looked up so the Actor always returns something, and every row records what was looked up in its `input` column.

#### Examples

Monthly snapshots of a page over two years:

```json
{ "urls": ["https://www.nytimes.com/"], "from": "2023", "to": "2024", "collapse": "month", "order": "oldest" }
```

Every page ever archived under a site's blog:

```json
{ "urls": ["example.com/blog/*"], "mode": "urls", "maxResultsPerUrl": 2000 }
```

Captures of one page that recorded an error (4xx or 5xx), newest first:

```json
{ "urls": ["example.com"], "statusFilter": "error", "collapse": "none" }
```

In URL-list mode every URL row already carries `statusCodesSeen`, so pages that ever returned a 404 can be picked out of the list directly.

### Output

Snapshot row:

```json
{
  "rowType": "snapshot",
  "input": "apify.com",
  "url": "https://apify.com/",
  "capturedAt": "2026-09-07T22:14:23Z",
  "timestamp": "20260907221423",
  "statusCode": 200,
  "statusClass": "ok",
  "captureType": "capture",
  "mimeType": "text/html",
  "lengthBytes": 73407,
  "digest": "FLYN6YGUV5ZLNCIBCXESIGJ43HLXV6NL",
  "sameContentAsPrevious": false,
  "archiveUrl": "https://web.archive.org/web/20260907221423/https://apify.com/",
  "rawArchiveUrl": "https://web.archive.org/web/20260907221423id_/https://apify.com/"
}
```

URL-list row (a real row, `docs.apify.com` with scope `host`, 2026-09-10):

```json
{
  "rowType": "url",
  "input": "docs.apify.com",
  "url": "https://docs.apify.com/",
  "urlKey": "com,apify,docs)/",
  "firstCapturedAt": "2019-12-11T00:41:01Z",
  "lastCapturedAt": "2026-08-21T23:05:38Z",
  "captureCount": 200,
  "statusCodesSeen": [200, 301, 403],
  "statusCode": 200,
  "mimeType": "text/html",
  "archiveUrl": "https://web.archive.org/web/20260821230538/https://docs.apify.com/",
  "firstArchiveUrl": "https://web.archive.org/web/20191211004101/https://docs.apify.com/",
  "complete": true
}
```

The archive also holds malformed addresses its crawler picked up - on docs.apify.com, for example, paths with fragments of JSON stuck to them. They are returned exactly as the archive records them, usually with a 404 in `statusCodesSeen`, so they are easy to filter out.

`lengthBytes` is the size of the stored (compressed) capture, as the archive reports it. `sameContentAsPrevious` compares the capture with the one immediately before it in time; with `collapse: month` that means the previous month's capture.

Rows with `status` other than `ok` (`no-captures`, `no-matching-status`, `bad-input`, `refused-by-archive`, `failed`, `incomplete`, `skipped-time-limit`) explain themselves in `note` and are never charged. `skipped-time-limit` means the entry was not answered because the run was about to reach its time limit (it was not looked up, or its retries were cut short) - run it again or give the run a longer timeout.

**Which capture represents a period.** With one per month, day or year, newest-first order returns the latest capture of each period and oldest-first the earliest (for apify.com in June 2026: 06-28 newest-first, 06-07 oldest-first). "Only when the content changed" always returns the capture where the change happened: in newest-first order the archive itself would return the last capture of each unchanged run, so this Actor reads up to the newest 10,000 captures un-thinned and picks the change points itself.

### Pricing

Pay per result: you are charged for each snapshot row or URL row returned. Rows that only explain an empty, rejected or failed lookup are free.

### Source

Data comes from the Internet Archive's public Wayback CDX server (`web.archive.org/cdx/search/cdx`). This Actor is not affiliated with the Internet Archive.

# Actor input Schema

## `urls` (type: `array`):

Pages or domains to look up, such as apify.com, https://www.nytimes.com/section/technology or docs.python.org. In URL-list mode you can also write *.example.com (every subdomain) or example.com/blog/* (everything under a path). A malformed entry is reported as its own row and never sent to the archive, because the archive would silently answer it with captures of a different address.

## `mode` (type: `string`):

Snapshots: one row per archived capture of each URL, with its date, HTTP status, content type, size and a link to the archived copy. URL list: one row per distinct URL the archive holds under a domain, host or path, with when it was first and last captured, how many times, and every status code it has returned.

## `matchType` (type: `string`):

Only used for the URL list. Domain covers the host and all its subdomains, host covers exactly that host, and prefix covers everything under the path you give.

## `from` (type: `string`):

Only captures on or after this date. YYYY, YYYY-MM, YYYY-MM-DD or the archive's own digits (for example 2021 or 20210315). An impossible date is rejected with a reason rather than sent - the archive answers one with an empty result that looks exactly like 'nothing archived'.

## `to` (type: `string`):

Only captures on or before this date, in the same forms as From date. The range is inclusive: 2021 means up to the end of 2021.

## `collapse` (type: `string`):

Snapshots only. The archive often holds many captures a day of the same page. Month keeps one capture per month (the latest of each month in newest-first order, the earliest in oldest-first), change keeps only the captures where the content changed, none returns every capture. For the most heavily archived pages (such as google.com) thinning out can time out at the archive; a narrower date range or none works there.

## `order` (type: `string`):

Snapshots only. Newest returns the most recent captures first; oldest starts from the first capture the archive holds.

## `statusFilter` (type: `string`):

Snapshots only. Keep captures whose recorded HTTP status is a success (2xx), a redirect (3xx) or an error (4xx/5xx). The archive applies this filter before thinning out, so with one-per-month you get one matching capture for each month rather than losing the months whose first capture was a redirect. About one capture in eight is a revisit that records no status at all; those only appear when this is set to any.

## `maxResultsPerUrl` (type: `integer`):

The most snapshot rows, or distinct URLs in URL-list mode, returned for each entry. Up to 10,000.

## Actor input object example

```json
{
  "urls": [
    "apify.com"
  ],
  "mode": "snapshots",
  "matchType": "domain",
  "from": "",
  "to": "",
  "collapse": "month",
  "order": "newest",
  "statusFilter": "any",
  "maxResultsPerUrl": 100
}
```

# Actor output Schema

## `results` (type: `string`):

One row per archived capture (snapshot mode) or per archived URL (URL-list mode), plus one row when an input was rejected, had no captures, or could not be read.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("neverempty/wayback-machine-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("neverempty/wayback-machine-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "apify.com"
  ]
}' |
apify call neverempty/wayback-machine-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neverempty/wayback-machine-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vgy4NUZbRHrOOMGwP/builds/WnvanmpKzlmhnf7Sl/openapi.json