# Wayback Machine Scraper: Historical URLs & Snapshots (`arman-bd/wayback-machine-scraper`) Actor

Query the Internet Archive CDX API: every archived URL for a domain with timestamps, status codes, MIME types and snapshot links. No login, no proxy, no browser.

- **URL**: https://apify.com/arman-bd/wayback-machine-scraper.md
- **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community)
- **Categories:** Developer tools, SEO tools, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 snapshot scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wayback Machine Scraper: Historical URLs & Snapshots

![Wayback Machine Scraper: Every capture the Internet Archive holds for a domain, original URL, timestamp, status, MIME type and a permanent replay link](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/wayback-machine-scraper.jpg)

**Wayback Machine Scraper** queries the **Internet Archive CDX index** and returns every capture it holds for a domain or URL prefix, original URL, capture timestamp, archived HTTP status, MIME type, content digest, byte length and a permanent replay link.

The CDX index is the Wayback Machine's raw capture ledger. It is the fastest way to answer "what URLs has this site ever had?" without crawling the live site: **no browser, no proxies, no login.** A domain with two decades of history returns in a handful of paginated requests.

**Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/wayback-machine-scraper.md)**

```
https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/wayback-machine-scraper.md
```

### What you get

| Output field | Meaning |
|---|---|
| `originalUrl` | The URL as it was crawled at capture time |
| `timestamp` | Raw Wayback timestamp, `YYYYMMDDhhmmss` in UTC |
| `snapshotDate` | The same instant as an ISO-8601 string |
| `archivedUrl` | Permanent replay link, `https://web.archive.org/web/<timestamp>/<url>` |
| `statusCode` | HTTP status the archive recorded, or `null` for revisit records |
| `mimeType` | Content type as served at capture time |
| `digest` | SHA-1 content digest, identical digests mean identical bytes |
| `length` | Compressed capture size in bytes |
| `urlkey` | Canonical SURT key the archive sorts on (`com,apify)/blog`) |
| `scrapedAt` | Run timestamp |

A `RUN_SUMMARY` record in the key-value store holds per-run counts (`snapshotsSaved`, `duplicatesSkipped`), the filters used, any target that failed, and any target skipped because `maxResults` was already reached.

### Common use cases

- **Recover deleted pages.** Find the last good capture of a page that no longer exists and pull it from the replay link.
- **SEO migration audits.** Enumerate every URL a site ever published, then diff against the new sitemap to find what you forgot to redirect.
- **Attack-surface discovery.** Historical hostnames and paths often expose staging, admin and API endpoints that are still live.
- **Competitor change tracking.** Collapse on `digest` and every row is a real content change, not a re-capture.
- **Link-rot repair.** For any dead outbound link, resolve the newest 200 capture and rewrite the reference.

### Quick start

Everything the archive has for a site, one row per unique URL:

```json
{
 "urls": ["apify.com"],
 "matchType": "domain",
 "collapse": "urlkey",
 "maxResults": 5000
}
```

Successful HTML pages from one section, in a date window:

```json
{
 "urls": ["apify.com/blog"],
 "matchType": "prefix",
 "fromDate": "20220101",
 "toDate": "20241231",
 "filterStatus": ["200"],
 "collapse": "urlkey",
 "maxResults": 2000
}
```

Change detection on a single page, one row per distinct version:

```json
{
 "urls": ["https://example.com/pricing"],
 "matchType": "exact",
 "collapse": "digest",
 "maxResults": 500
}
```

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `urls` | array | - | **Required.** Hosts, URL prefixes or full URLs. Schemes and trailing slashes are stripped automatically. |
| `matchType` | string | `prefix` | `exact`, `prefix`, `host` or `domain`. See the table below. |
| `fromDate` | string | `""` | Earliest capture, `YYYYMMDD` (`YYYY` and `YYYYMM` also work). |
| `toDate` | string | `""` | Latest capture, same format. |
| `filterStatus` | array | `[]` | Keep only these archived HTTP status codes. Empty = every capture. |
| `collapse` | string | `urlkey` | Collapse consecutive rows sharing a field. Empty = every raw capture. |
| `maxResults` | integer | `1000` | Hard cap on the total rows saved, split evenly between targets. Budget one target does not spend rolls to the next. `0` = no limit. |

#### Match types

| `matchType` | For `apify.com/blog` it returns |
|---|---|
| `exact` | Only `apify.com/blog` |
| `prefix` | `apify.com/blog` and everything beneath it |
| `host` | Every path on `apify.com` |
| `domain` | Every path on `apify.com` **and** every subdomain |

#### Collapse values

| `collapse` | Effect |
|---|---|
| `urlkey` | One row per unique URL, the right default for URL discovery |
| `timestamp:8` | One row per URL per day, good for change frequency |
| `digest` | Drops re-captures of unchanged bytes, one row per real content change |
| *(empty)* | Every raw capture, including thousands of identical ones |

Collapsing happens on *consecutive* rows in the archive's sort order, so `urlkey` and `digest` are reliable while an arbitrary field is not.

### Output example

```json
{
 "originalUrl": "https://apify.com/",
 "timestamp": "20200109130705",
 "archivedUrl": "https://web.archive.org/web/20200109130705/https://apify.com/",
 "statusCode": 200,
 "mimeType": "text/html",
 "digest": "D4P2GPUK3PEGEFYQFUENG7MNTQI3ZRN4",
 "length": 29306,
 "urlkey": "com,apify)/",
 "snapshotDate": "2020-01-09T13:07:05Z",
 "scrapedAt": "2026-08-06T11:44:53.773Z"
}
```

### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/arman-bd~wayback-machine-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "urls": ["apify.com"],
 "matchType": "domain",
 "filterStatus": ["200"],
 "maxResults": 500
 }'
```

### JavaScript example

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/wayback-machine-scraper').call({
 urls: ['apify.com/blog'],
 matchType: 'prefix',
 fromDate: '20230101',
 filterStatus: ['200'],
 maxResults: 1000,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const snap of items) console.log(`${snap.snapshotDate} ${snap.statusCode} ${snap.originalUrl}`);
```

### Notes

- **Columns are read from the header row.** CDX returns an array of arrays whose first row names the columns. This Actor builds its field map from that row on every response, so an archive-side column reorder cannot silently shift your data.
- **Pagination uses resume keys.** Each request asks for `showResumeKey=true` and feeds the returned cursor into the next call. If the archive ever hands back a cursor it already gave, the Actor stops and logs a warning instead of looping.
- **Identical rows are saved once per run.** The index returns the same capture more than once — through two targets that overlap, and within a single uncollapsed target. Any row identical to one already saved (same URL, timestamp, digest, status, MIME type, byte length and sort key) is dropped before it reaches the dataset and counted in `RUN_SUMMARY.duplicatesSkipped`. A row that differs in any of those fields is a different index record and is kept.
- **Bad targets don't kill the run.** A target with no captures returns an empty list and is logged as a warning; a target that errors is recorded in `RUN_SUMMARY.failures`. The Actor only throws if *every* target fails.
- **Long timeouts on purpose.** CDX is a cold index scan, not a search engine, wide `domain` queries routinely take 30-60 s. Requests allow 120 s and retry 429/5xx with exponential backoff.
- **Public data only.** No authentication, no personal data, no access-control bypass.

### Limits and behaviour

- **Always set `collapse` and `maxResults` on a big domain.** An uncollapsed `matchType=domain` query on a large site can enumerate millions of captures and will exhaust the run timeout before it finishes.
- **`maxResults` is a hard ceiling on the run, not a per-target allowance.** The budget is recomputed from what has actually been saved, so ten targets with `maxResults: 10` finish at ten rows, and a target that matches only a few leaves the rest of its share to the targets after it. Targets the budget never reached are listed in `RUN_SUMMARY.targetsSkipped`.
- **The archive throttles under load.** Retries are exponential (2 s, 4 s, 8 s) because linear backoff does not clear archive rate limits.
- **`statusCode` is `null` for revisit records.** The archive writes `-` where a capture is a pointer to identical earlier bytes; this Actor emits `null` rather than a fake code.

### FAQ

**Do I need a proxy?** No. Proxy configuration is not required to run this Actor.

**Do I need an Internet Archive account?** No. You supply no credentials.

**Can I get the page content itself?** Not directly, this Actor returns the index. `archivedUrl` is a permanent replay link you can fetch for the bytes.

**Why did my query return nothing?** Usually `matchType`. `exact` on `apify.com` matches only the bare homepage URL; you probably want `prefix` or `host`. Check `RUN_SUMMARY.filters` for what actually ran.

**Why does the same URL appear many times?** Each row is one capture, so an uncollapsed query returns every visit the archive ever made to that URL. Set `collapse` to `urlkey` for one row per URL, or `timestamp:8` for one per URL per day. Rows that are *identical* — same URL, timestamp, digest, status, MIME type, byte length and sort key — are a different matter: the index does repeat those, and the Actor drops the repeat before saving it, so you are never billed twice for the same row. The count is in `RUN_SUMMARY.duplicatesSkipped`. Two rows sharing a URL and timestamp but differing in status or MIME type are genuinely two index records, usually an original capture plus a `warc/revisit` pointer, and both are kept.

**Why did I get fewer rows than `maxResults`?** `maxResults` is a ceiling, not a quota. The archive may simply hold fewer matching captures, and identical rows dropped as duplicates do not consume it. If targets at the end of your list were never queried because the budget ran out, they are listed in `RUN_SUMMARY.targetsSkipped`.

**What happens if the archive is unavailable?** The target is retried with backoff, then recorded in `RUN_SUMMARY.failures`; the run continues with the remaining targets.

**Can I schedule it?** Yes, it is designed for scheduled runs. Diff on `digest` between runs to detect real content changes.

**Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

# Actor input Schema

## `urls` (type: `array`):

Hosts (apify.com), URL prefixes (apify.com/blog) or full URLs. Schemes and trailing slashes are stripped automatically. How each entry is expanded depends on 'Match type'.

## `matchType` (type: `string`):

exact = that one URL only. prefix = the URL and everything beneath it. host = every path on that exact host. domain = the host plus all its subdomains (widest, slowest).

## `fromDate` (type: `string`):

Earliest capture to return, as YYYYMMDD (or YYYY, or YYYYMM). Leave empty to start at the archive's first capture.

## `toDate` (type: `string`):

Latest capture to return, as YYYYMMDD (or YYYY, or YYYYMM). Leave empty to run up to the most recent capture.

## `filterStatus` (type: `array`):

Keep only captures whose archived response had one of these status codes. Leave empty to keep every capture, including redirects and errors.

## `collapse` (type: `string`):

Collapse consecutive rows sharing a field value. 'urlkey' gives one row per unique URL, 'timestamp:8' one row per URL per day, 'digest' drops unchanged re-captures. Leave empty for every raw capture.

## `maxResults` (type: `integer`):

Cap the total number of snapshots saved across all targets. The budget is split evenly between targets. Set 0 for no limit. only do that on narrow queries.

## Actor input object example

```json
{
  "urls": [
    "apify.com",
    "https://apify.com/store"
  ],
  "matchType": "domain",
  "fromDate": "20200101",
  "toDate": "20241231",
  "filterStatus": [
    "200",
    "301"
  ],
  "collapse": "timestamp:8",
  "maxResults": 1000
}
```

# Actor output Schema

## `items` (type: `string`):

Every record the run produced.

## `runsummary` (type: `string`):

The RUN\_SUMMARY record from the run's key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arman-bd/wayback-machine-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("arman-bd/wayback-machine-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "apify.com"
  ]
}' |
apify call arman-bd/wayback-machine-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/wayback-machine-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nHhqSUw05ja3VcVSE/builds/Eyk2V7qNR1TkopNw0/openapi.json