# Wayback Machine URL Archive Scraper (`gochujang/wayback-machine-scraper`) Actor

Search and retrieve archived snapshots of any URL from Internet Archive's Wayback Machine. Supports date range filtering, deduplication, status code filtering, and optional HTML content fetching via CDX API. Ideal for web archive research, deleted content recovery, and SEO history analysis.

- **URL**: https://apify.com/gochujang/wayback-machine-scraper.md
- **Developed by:** [Hojun Lee](https://apify.com/gochujang) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wayback Machine URL Archive Scraper

Search and retrieve archived snapshots of any URL from the [Internet Archive's Wayback Machine](https://web.archive.org/) via its CDX API. Batch multiple URLs, filter by date range or HTTP status, deduplicate results, and optionally fetch the archived HTML content.

***

### What It Does

This Actor queries the [CDX Server API](https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md) — a fast index of the Wayback Machine — to retrieve metadata and optionally content for every archived snapshot of one or more URLs.

Each output item represents a single archived snapshot including its timestamp, status code, MIME type, content digest, and a direct playback URL.

***

### Features

- **Batch URL support** — process multiple URLs in a single run with controlled concurrency
- **Date range filtering** — restrict results to a specific time window (YYYYMMDD format)
- **Deduplication** — collapse by content digest, day, or month to avoid redundant snapshots
- **HTTP status filtering** — limit results to 200 OK, 301 redirects, or any combination
- **Latest-only mode** — return only the most recent snapshot per URL
- **Optional HTML content fetch** — retrieve the full archived HTML for each snapshot
- **PPE pricing** — pay only for what you use ($0.001 per snapshot returned)

***

### Input Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `urls` | array of strings | **required** | URLs to search in the Wayback Machine |
| `dateFrom` | string | — | Start date `YYYYMMDD` (e.g. `20200101`). No lower bound if omitted. |
| `dateTo` | string | — | End date `YYYYMMDD` (e.g. `20241231`). No upper bound if omitted. |
| `limit` | integer | `100` | Max snapshots per URL (1–1000) |
| `collapseBy` | enum | `digest` | Deduplication strategy: `digest`, `timestamp:8` (daily), `timestamp:6` (monthly), or `""` (none) |
| `statusFilter` | string | `200` | HTTP status codes to include. `"200"`, `"200,301"`, or `""` for all |
| `includeLatestOnly` | boolean | `false` | Return only the most recent snapshot per URL |
| `fetchContent` | boolean | `false` | Fetch archived HTML content for each snapshot (increases runtime and cost) |

#### Example Input

```json
{
  "urls": [
    "https://example.com",
    "https://github.com/openai"
  ],
  "dateFrom": "20200101",
  "dateTo": "20241231",
  "limit": 100,
  "collapseBy": "digest",
  "statusFilter": "200",
  "includeLatestOnly": false,
  "fetchContent": false
}
```

***

### Output Fields

Each item in the dataset represents one archived snapshot.

| Field | Type | Description |
|-------|------|-------------|
| `original_url` | string | The original URL that was archived |
| `timestamp` | string | CDX timestamp in `YYYYMMDDHHmmSS` format |
| `datetime` | string | ISO 8601 UTC datetime (e.g. `2024-01-15T12:34:56Z`) |
| `snapshot_url` | string | Direct Wayback Machine playback URL |
| `status_code` | string | HTTP status code of the archived response |
| `mime_type` | string | MIME type of the archived content |
| `digest` | string | SHA1 content digest (prefixed with `SHA1:`) |
| `length` | integer | Size of the archived content in bytes |
| `content` | string or null | Full archived HTML content (only when `fetchContent: true`) |

#### Example Output Item

```json
{
  "original_url": "https://example.com",
  "timestamp": "20240115123456",
  "datetime": "2024-01-15T12:34:56Z",
  "snapshot_url": "https://web.archive.org/web/20240115123456/https://example.com",
  "status_code": "200",
  "mime_type": "text/html",
  "digest": "SHA1:ABCDEF1234567890ABCDEF1234567890ABCDEF12",
  "length": 12345,
  "content": null
}
```

***

### Use Cases

#### Web Archive Research

Track how a website evolved over time. Useful for journalism, academic research, and competitive intelligence.

#### Deleted Content Recovery

Find previously published pages, articles, or product listings that have since been removed from the live web.

#### SEO History Analysis

Audit historical title tags, meta descriptions, and page structures to understand past SEO strategies and content changes.

#### Competitive Monitoring

Build a timeline of a competitor's landing pages, pricing pages, or product launches using archived snapshots.

#### Legal & Compliance Evidence

Retrieve timestamped proof of past web content for legal documentation or intellectual property disputes.

***

### Pricing

This Actor uses **Pay-Per-Event (PPE)** pricing:

| Event | Price |
|-------|-------|
| Actor start | $0.005 (one-time per run) |
| Per snapshot returned | $0.001 |

**Example:** Searching 5 URLs and retrieving 200 snapshots total costs approximately $0.205.

***

### Technical Details

- Built with Python + `httpx` (async HTTP client)
- Queries the [Internet Archive CDX API](http://web.archive.org/cdx/search/cdx)
- Processes up to 3 URLs concurrently (configurable via semaphore)
- No external proxy or third-party service required — all data comes directly from `web.archive.org`

***

### Related actors

- [Wikipedia Article Scraper](https://apify.com/gochujang/wikipedia-article-scraper) — Wikipedia revision history as an alternative archive source for content drift analysis
- [Domain DNS Checker](https://apify.com/gochujang/domain-dns-checker) — DNS records for archived domains to verify current registration status
- [Open Library Search](https://apify.com/gochujang/open-library-search) — Internet Archive's book catalog alongside web archive snapshots for full research workflows

### Feedback

If this actor helps your web archival research, a review helps others find it: [Leave a review on Apify Store](https://apify.com/gochujang/wayback-machine-scraper#reviews)

# Actor input Schema

## `urls` (type: `array`):

List of URLs to search in the Wayback Machine CDX API.

## `dateFrom` (type: `string`):

Start date for snapshot search in YYYYMMDD format (e.g. 20200101). Leave empty for no lower bound.

## `dateTo` (type: `string`):

End date for snapshot search in YYYYMMDD format (e.g. 20241231). Leave empty for no upper bound.

## `limit` (type: `integer`):

Maximum number of snapshots to return per URL.

## `collapseBy` (type: `string`):

Deduplicate results by this field. 'digest' returns one snapshot per unique content hash; 'timestamp:8' collapses by day; 'timestamp:6' by month; empty string returns all.

## `statusFilter` (type: `string`):

Filter by HTTP status code(s). Use '200' for OK only, '200,301' for multiple, or leave empty for all status codes.

## `includeLatestOnly` (type: `boolean`):

If true, return only the most recent snapshot for each URL.

## `fetchContent` (type: `boolean`):

If true, fetch the full HTML content of each archived snapshot. Increases runtime and cost.

## Actor input object example

```json
{
  "urls": [
    "https://example.com"
  ],
  "limit": 100,
  "collapseBy": "digest",
  "statusFilter": "200",
  "includeLatestOnly": false,
  "fetchContent": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("gochujang/wayback-machine-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("gochujang/wayback-machine-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call gochujang/wayback-machine-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gochujang/wayback-machine-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BJMjgCxxFfade5IQI/builds/c5vFQYXF3UEke4Pxd/openapi.json