# Wayback Machine Snapshot History Scraper (`automation-lab/wayback-machine-snapshot-history-scraper`) Actor

Export Internet Archive Wayback Machine snapshot history with replay URLs, timestamps, status, MIME, digest, and size filters.

- **URL**: https://apify.com/automation-lab/wayback-machine-snapshot-history-scraper.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wayback Machine Snapshot History Scraper

Export Internet Archive CDX snapshot histories for domains, hosts, URL prefixes, or exact URLs.

Get capture timestamps, original URLs, HTTP statuses, MIME types, content digests, byte lengths, and ready-to-open replay URLs in a structured Apify dataset.

The Actor uses the public Wayback Machine CDX API directly. It needs no browser, login, API token, cookie, or proxy.

### What does this Wayback Machine scraper do?

Wayback Machine Snapshot History Scraper turns Internet Archive CDX metadata into analysis-ready records.

Use it to:

- inventory archived URLs before a site migration;
- find historical versions of a page;
- build redirect and content-recovery maps;
- compare digests to identify changed captures;
- document when public web content appeared or changed;
- schedule recurring archival checks;
- export replay links for review by researchers.

This Actor exports capture metadata. It does not download or parse the archived page body.

### Who is it for?

**SEO and migration teams** can recover old URL inventories and plan redirects.

**Site owners** can locate recoverable versions of lost pages.

**OSINT and legal researchers** can create dated, source-linked capture timelines.

**Analysts and journalists** can monitor historical changes to public pages.

**Developers** can feed normalized archive metadata into databases, spreadsheets, or change-detection pipelines.

### Why use this Actor?

- Multiple input scopes: domain, host, URL prefix, or exact URL.
- Date, HTTP status, and MIME type filters are applied by CDX.
- Optional digest collapse removes unchanged duplicate captures.
- Every row includes a direct replay URL.
- Pagination respects a run-wide maximum item count.
- Transient timeouts, 429 responses, and 5xx errors use bounded backoff.
- Partial rows already saved remain available if a later request fails.
- Typed dataset views work with Apify integrations and API clients.
- Direct public HTTP keeps memory and infrastructure cost low.

### What data can I extract?

| Field | Meaning |
| --- | --- |
| `sourceQuery` | The input target that produced the record |
| `captureTimestamp` | CDX UTC timestamp in `YYYYMMDDhhmmss` format |
| `captureDate` | The same capture time as ISO 8601 |
| `originalUrl` | URL recorded by Internet Archive |
| `statusCode` | Archived HTTP response status, when available |
| `mimeType` | Archived response MIME type, when available |
| `digest` | Internet Archive content digest for change analysis |
| `byteLength` | CDX compressed record length, when available |
| `replayUrl` | Direct Wayback replay URL using `id_` mode |
| `cdxApiUrl` | Exact CDX API page request used by the Actor |
| `fetchedAt` | Time when the metadata page was fetched |

Fields unavailable in an upstream CDX row are returned as `null`.

### How to scrape Wayback Machine snapshot history

1. Open the Actor input form.
2. Add one or more domains, URLs, or patterns to **Targets**.
3. Select the match type that describes the intended scope.
4. Optionally set `from`, `to`, status, MIME, and digest-collapse filters.
5. Choose a practical `maxItems` limit.
6. Click **Start**.
7. Open the **Snapshot history** dataset view.
8. Export records as JSON, CSV, Excel, XML, or RSS.

Start with a narrow exact URL or date range before requesting a large domain.

### Input parameters

#### `targets`

Required list of 1–100 domains, URLs, or URL patterns.

Examples:

- `nasa.gov`
- `https://www.whitehouse.gov/`
- `https://docs.python.org/3/`

#### `matchType`

- `exact`: only the supplied URL;
- `prefix`: URLs beginning with the supplied prefix;
- `host`: one hostname;
- `domain`: a domain and its subdomains.

The default is `domain`.

#### `from` and `to`

Optional UTC capture boundaries using 4–14 digits.

Examples include `2020`, `20200101`, and `20200101120000`.

#### `statusCodes`

Optional exact codes such as `200` and `301`, or classes such as `2..`.

#### `mimeTypes`

Optional MIME filters such as `text/html`, `application/pdf`, or `image/*`.

#### `collapseByDigest`

When enabled, CDX collapses repeated records with the same content digest.

#### `maxItems`

Run-wide maximum across every target, from 1 to 100,000.

#### `pageSize`

CDX page size from 1 to 2,000. The default is 500.

#### `maxRetries`

Retry count for transient failures, from 0 to 8. The default is 4.

### Example input

```json
{
  "targets": ["https://www.whitehouse.gov/"],
  "matchType": "exact",
  "from": "20200101",
  "to": "20201231",
  "statusCodes": ["200"],
  "mimeTypes": ["text/html"],
  "collapseByDigest": true,
  "maxItems": 250
}
```

### Example output

```json
{
  "sourceQuery": "https://www.whitehouse.gov/",
  "captureTimestamp": "20200101034743",
  "captureDate": "2020-01-01T03:47:43Z",
  "originalUrl": "https://www.whitehouse.gov/",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "5U4R6QJEOVRLUWTNMLD6NNM6HL64OKLI",
  "byteLength": 16014,
  "replayUrl": "https://web.archive.org/web/20200101034743id_/https://www.whitehouse.gov/",
  "cdxApiUrl": "https://web.archive.org/cdx/search/cdx?...",
  "fetchedAt": "2026-07-27T03:15:00.000Z"
}
```

This example reflects real CDX output. `fetchedAt` varies by run.

### How much does it cost to export Wayback Machine captures?

The Actor uses pay-per-event pricing: one **Run started** event per run and one **Item extracted** event per saved capture.

The BRONZE price is $0.004 per run plus $0.000035714 per capture. Higher subscription tiers receive progressively lower per-item prices as shown in the Apify pricing tab.

At the BRONZE item price:

- 10 captures cost about $0.00436 including the start event;
- 100 captures cost about $0.00757 including the start event;
- 1,000 captures cost about $0.03971 including the start event.

You are never charged an item event for a rejected, duplicate, empty, or failed row. Apify displays the active tier and final run charge before billing.

### SEO migration and redirect-map workflow

1. Query the old domain with `matchType: domain`.
2. Restrict dates to the period before migration.
3. Select successful HTML captures.
4. Enable digest collapse to reduce unchanged duplicates.
5. Export `originalUrl`, `captureDate`, `digest`, and `replayUrl`.
6. Group by original URL.
7. Review the newest replay for each old URL.
8. Map recovered URLs to current destinations.

For large sites, split work by hostname, path prefix, or year.

### Historical change timeline workflow

Schedule the Actor with an exact URL and digest collapse enabled.

Store rows in Google Sheets, Airtable, a webhook receiver, or your data warehouse.

A new digest can indicate changed archived content. Digest equality is a useful signal, but it is not a semantic comparison of rendered pages.

### Export and integrations

The default dataset supports:

- JSON, JSONL, CSV, Excel, XML, and RSS exports;
- Google Sheets and Zapier integrations;
- Make workflows;
- webhooks after a successful run;
- JavaScript and Python API clients;
- direct dataset API access;
- Apify schedules for repeat monitoring.

Use `sourceQuery` to separate records when one run contains several targets.

### Run with the Apify API using cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~wayback-machine-snapshot-history-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"targets":["nasa.gov"],"matchType":"domain","statusCodes":["200"],"mimeTypes":["text/html"],"collapseByDigest":true,"maxItems":100}'
```

The response contains the run ID and default dataset ID.

### Run with JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/wayback-machine-snapshot-history-scraper').call({
  targets: ['https://docs.python.org/3/'],
  matchType: 'prefix',
  statusCodes: ['2..'],
  mimeTypes: ['text/html'],
  collapseByDigest: true,
  maxItems: 1000,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

### Run with Python

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/wayback-machine-snapshot-history-scraper').call(run_input={
    'targets': ['https://www.whitehouse.gov/'],
    'matchType': 'exact',
    'from': '20200101',
    'to': '20201231',
    'statusCodes': ['200'],
    'mimeTypes': ['text/html'],
    'maxItems': 250,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)
```

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/wayback-machine-snapshot-history-scraper"
```

#### Claude Desktop, Cursor, and VS Code setup

Claude Desktop, Cursor, and VS Code can use this MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/wayback-machine-snapshot-history-scraper"
    }
  }
}
```

Example prompts:

- “Export successful NASA HTML snapshots captured between 2018 and 2020.”
- “Find digest-distinct versions of the White House homepage during 2020.”
- “Build a historical URL inventory below the Python 3 docs prefix.”

### Limits and responsible scaling

Internet Archive is a public service and may throttle broad or frequent queries.

Keep `pageSize` and concurrency conservative. This Actor processes targets sequentially.

A domain query may represent millions of records. Use date ranges, path prefixes, filters, and `maxItems` to make work bounded.

The Actor retries temporary failures but does not retry malformed input or deterministic 4xx responses.

No automatic proxy fallback is used.

### Data freshness and interpretation

CDX describes captures indexed by Internet Archive; it is not a live crawl of the target site.

A missing result does not prove that a page never existed.

Indexing delays, exclusions, takedowns, robots policies, and archive availability can affect results.

A replay URL may be unavailable even when a CDX record exists.

`byteLength` is CDX metadata and should not be treated as the uncompressed page size.

### Legal and responsible use

Use archived public-web metadata for lawful purposes.

Respect Internet Archive terms, source-site rights, privacy obligations, and applicable retention or evidence rules.

Do not use historical pages to expose sensitive personal data, evade access controls, or misrepresent archived content as current.

For legal or evidentiary work, preserve run input, timestamps, dataset exports, and independent chain-of-custody records. This Actor does not certify authenticity.

### Troubleshooting

#### Why did the dataset contain no records?

Check the target spelling, `matchType`, date range, status filters, and MIME filters. Try an exact known URL with fewer filters.

#### Why did the run report HTTP 429 or a timeout?

Reduce `pageSize`, narrow the date or URL scope, and retry later. The Actor already applies bounded exponential backoff.

#### Why are there many similar captures?

Enable `collapseByDigest`. Separate captures can still share similar content while having different digests.

#### Why are some metadata values null?

Older or unusual CDX records may omit a status, MIME type, digest, or length.

#### Why does a replay link fail?

Archive replay availability can differ from CDX indexing. Try the standard Wayback calendar for the original URL.

### FAQ

#### Does it download archived page content?

No. It exports snapshot metadata and replay links only.

#### Does it require an Internet Archive account?

No. It uses the anonymous public CDX endpoint.

#### Does it use an Apify proxy?

No. Requests are direct, so users do not incur proxy transfer for this Actor.

#### Can it find every URL ever archived for a domain?

It can query domain scope, but complete histories can be very large and upstream availability is not guaranteed. Use bounded filters and limits.

#### Can I monitor several targets in one run?

Yes. Add up to 100 targets. `maxItems` applies across the whole run.

#### How do I detect changes?

Enable digest collapse or compare `digest` values over `captureDate`. For semantic differences, fetch authorized replay pages in a downstream process.

### Related automation-lab Actors

This Actor is intentionally standalone because no active automation-lab Actor currently provides the same Wayback CDX snapshot-history workflow.

Combine its dataset with your existing URL audit, redirect mapping, spreadsheet, webhook, or data-warehouse automation.

Only automation-lab Actors will be linked here when a genuinely complementary published product is available.

# Actor input Schema

## `targets` (type: `array`):

One to 100 targets. Examples: nasa.gov, https://www.nasa.gov/history/, or nasa.gov/\*.

## `matchType` (type: `string`):

Domain includes subdomains; host stays on one host; prefix finds URLs below a prefix; exact matches one URL.

## `from` (type: `string`):

Optional UTC boundary using 4–14 digits, from YYYY to YYYYMMDDhhmmss.

## `to` (type: `string`):

Optional inclusive UTC boundary using 4–14 digits, from YYYY to YYYYMMDDhhmmss.

## `statusCodes` (type: `array`):

Exact status codes (200, 301) or classes (2.., 3..). All selected filters must match.

## `mimeTypes` (type: `array`):

CDX MIME types such as text/html, application/pdf, or image/\*.

## `collapseByDigest` (type: `boolean`):

Return only one capture for repeated records with the same Internet Archive digest.

## `maxItems` (type: `integer`):

Maximum dataset records across all targets.

## `pageSize` (type: `integer`):

Records requested per CDX page. Lower this if the archive rate-limits large queries.

## `maxRetries` (type: `integer`):

Retries for timeouts, HTTP 429, and temporary 5xx responses.

## Actor input object example

```json
{
  "targets": [
    "nasa.gov"
  ],
  "matchType": "domain",
  "statusCodes": [
    "200"
  ],
  "mimeTypes": [
    "text/html"
  ],
  "collapseByDigest": false,
  "maxItems": 20,
  "pageSize": 500,
  "maxRetries": 4
}
```

# Actor output Schema

## `overview` (type: `string`):

Open the default dataset in the snapshot-history table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targets": [
        "nasa.gov"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/wayback-machine-snapshot-history-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "targets": ["nasa.gov"] }

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/wayback-machine-snapshot-history-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targets": [
    "nasa.gov"
  ]
}' |
apify call automation-lab/wayback-machine-snapshot-history-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=automation-lab/wayback-machine-snapshot-history-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/AOgv2ugBnEOnXu0Cb/builds/fqf0mYLhMGWEKgvys/openapi.json