# Wayback Machine Scraper: Snapshots, Archived URLs, History (`joesimo/wayback-machine`) Actor

Get Wayback Machine snapshots for any URL, list every archived URL of a domain, or get a first/last capture history summary, through the Internet Archive CDX API. Optional page text. No login. Empty or invalid inputs are free.

- **URL**: https://apify.com/joesimo/wayback-machine.md
- **Developed by:** [Joe Simo](https://apify.com/joesimo) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 snapshots

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Machine Scraper: Snapshots, Archived URLs, History

Pull data out of the Internet Archive Wayback Machine for a list of URLs or domains. Pick one of three outputs:

- **Snapshots**: one row per capture of a page, with the capture time, the HTTP status the site returned, MIME type, content digest and a ready Wayback link. Collapse to content changes only, or to the first capture per day, month or year. Optionally add the archived page title and visible text.
- **Archived URLs**: every distinct URL the Wayback Machine has stored under a domain, subdomain or path, with the date it was first captured. Good for finding old pages, removed product pages, lost blog posts and old sitemaps.
- **Site history summary**: one row per URL with the first and latest capture, days since the latest capture, how many months and years were archived, and a per-year month count.

Data comes from the public Wayback CDX API at web.archive.org. No login, no API key, no proxy, no browser.

Empty input runs a demo: one snapshot per year of apify.com (about 20 rows).

**Pricing (pay per event):** $0.0005 per snapshot row, $0.0005 per archived URL row ($0.50 per 1,000), $0.003 per site history summary, plus $0.002 per page when you turn on page text. URLs with no captures, failed lookups and invalid lines are saved as free status rows.

### What you get

Snapshot rows (`kind: snapshot`):

- `url` (as archived), `urlKey`, `timestamp` (YYYYMMDDhhmmss), `capturedAt` (ISO 8601 UTC), `year`
- `statusCode` (200, 301, 404 and so on, as the site answered the crawler), `mimeType`, `isRevisit` (same content as an earlier capture)
- `digest` (SHA-1 of the content: the same digest means the page did not change), `lengthBytes`
- `archiveUrl` (opens the capture in the Wayback Machine) and `rawArchiveUrl` (the original bytes without the Wayback toolbar)
- With page text on: `pageTitle`, `metaDescription`, `text` (visible text, scripts and styles removed), `textLength`, `textTruncated`, `textStatus`

Archived URL rows (`kind: url`): `url`, `urlKey`, `host`, `path` (with query string), `firstSeenAt`, `statusCode` and `mimeType` of that first capture, `archiveUrl`.

Summary rows (`kind: summary`): `firstCaptureAt`, `firstArchiveUrl`, `lastCaptureAt`, `lastArchiveUrl`, `lastStatusCode`, `daysSinceLastCapture`, `yearsArchived`, `monthsWithCaptures`, `monthsWithCapturesByYear`, `archivedSpanYears`.

### How to run it

| Goal | Input |
| --- | --- |
| Every capture of one page | `"urls": ["stripe.com/pricing"]` |
| Only captures where the page changed | `"urls": ["stripe.com/pricing"], "collapse": "digest"` |
| One capture per month in 2024 | `"collapse": "month", "dateFrom": "2024", "dateTo": "2024"` |
| Latest 50 captures | `"order": "newest", "maxResultsPerUrl": 50` |
| Old page text for content research | `"collapse": "year", "includePageText": true` |
| All archived URLs of a site and its subdomains | `"urls": ["example.com"], "mode": "urls"` |
| Archived URLs under one path | `"urls": ["example.com/blog/*"], "mode": "urls"` |
| Only archived PDFs of a site | `"mode": "urls", "mimeTypes": ["application/pdf"]` |
| First and last capture for a list of domains | `"urls": ["a.com", "b.com"], "mode": "summary"` |
| Only successful captures | `"statusCodes": ["200"]` (or `"!404"` to drop 404s) |

Input patterns: `example.com` and full URLs match that exact page (both http and https, with or without www as stored by the archive). `*.example.com` matches the domain and every subdomain. `example.com/blog/*` matches everything under that path. In archived URLs mode a bare domain matches the whole domain automatically. You can also set Match type by hand.

### Good uses

- SEO: find old URLs to redirect after a migration, recover lost content, check when a page first appeared
- Domain research: see how long a domain has been live and when it was last active before buying it
- Competitor tracking: list every capture where a pricing or terms page changed, with links to compare
- Compliance and legal: dated, linkable evidence of what a page showed on a given day
- Research datasets: page text from a fixed period, one capture per month or year

### How it works

The Actor calls the Wayback CDX search API with your URL, match type, date range, status and MIME filters and collapse setting, and pages through results with resume keys. For page text it fetches the raw archived HTML (`id_` URLs) of captures with status 200 and strips scripts, styles and tags.

The Internet Archive limits the CDX API to about 30 requests per minute per IP. This Actor sends one CDX request every 2.6 seconds (about 23 per minute), identifies itself with a descriptive User-Agent, and pauses 60 seconds when the archive answers HTTP 429. After three rate limit answers in a row it stops and saves a free `failed` row instead of pushing harder. Server errors (5xx) and timeouts are retried twice.

### Pricing

| Event | Price | When |
| --- | --- | --- |
| Snapshot | $0.0005 | One capture row saved (snapshots mode) |
| Archived URL | $0.0005 | One distinct archived URL saved (archived URLs mode) |
| Site history summary | $0.003 | One summary row for a URL with captures |
| Page text | $0.002 | Extra, per snapshot where page text was extracted |

Examples: 1,000 captures of a page cost $0.50. 1,000 captures with text cost $2.50. A history summary for 100 domains costs $0.30. Set "Max rows in total" or a maximum cost per run to cap spend; the Actor stops exactly at the cap.

### Input example

```json
{
    "urls": ["apify.com/pricing"],
    "mode": "snapshots",
    "collapse": "digest",
    "statusCodes": ["200"],
    "maxResultsPerUrl": 200
}
```

### Output example

```json
{
    "status": "ok",
    "kind": "snapshot",
    "input": "example.com",
    "matchType": "exact",
    "url": "https://www.example.com/",
    "urlKey": "com,example)/",
    "timestamp": "20241231223224",
    "capturedAt": "2024-12-31T22:32:24.000Z",
    "year": 2024,
    "statusCode": 200,
    "mimeType": "text/html",
    "isRevisit": false,
    "digest": "WJM2KPM4GF3QK2BISVUH2ASX64NOUY7L",
    "lengthBytes": 1351,
    "archiveUrl": "https://web.archive.org/web/20241231223224/https://www.example.com/",
    "rawArchiveUrl": "https://web.archive.org/web/20241231223224id_/https://www.example.com/",
    "pageTitle": "Example Domain",
    "text": "Example Domain\nThis domain is for use in illustrative examples in documents...",
    "textStatus": "ok"
}
```

Summary row:

```json
{
    "status": "ok",
    "kind": "summary",
    "input": "apify.com",
    "url": "http://www.apify.com:80/",
    "archived": true,
    "firstCaptureAt": "2007-05-31T10:15:38.000Z",
    "lastCaptureAt": "2026-11-04T21:52:54.000Z",
    "lastStatusCode": 200,
    "daysSinceLastCapture": 5,
    "yearsArchived": 18,
    "monthsWithCaptures": 128,
    "archivedSpanYears": 19.4
}
```

A run summary (rows per URL, CDX requests, rate limit pauses, stop reason) is saved as `OUTPUT` in the default key-value store.

### Limits and notes

- Speed is set by the archive's rate limit: about 23 CDX requests per minute. One exact-URL lookup usually takes 1 to 3 seconds; domain and prefix scans on large sites can take 10 to 60 seconds per page of results. Page text adds about 0.5 to 2 seconds per page.
- Newest first returns up to 10,000 captures per URL. Oldest first pages through all captures up to "Max rows per URL".
- `statusCode` is null for revisit records (the archive stored only a pointer to identical earlier content).
- Some sites asked the Internet Archive to exclude them; those return no captures (a free `empty` row).
- Archived pages keep the rights of their original owners. This Actor returns capture metadata, links and page text for research; check the original site's terms before reusing content.
- Not affiliated with or endorsed by the Internet Archive.

### Integrations and API

Run it from the Apify API, a schedule, Make, Zapier, n8n or another Actor, and read the dataset as JSON, CSV or Excel. Pass URLs in `urls` (string list) or `startUrls` (request list).

### FAQ

**Is this the official Wayback Machine API?** It uses the Internet Archive's public CDX search API, the same index behind the Wayback Machine calendar view.

**Why did a URL come back empty?** The archive has no capture matching the URL, match type and filters. Try `*.example.com` or a prefix pattern, or remove the status and date filters.

**Do I pay for empty results?** No. Empty, failed and invalid inputs are free status rows.

**Can it save full HTML or screenshots?** It saves visible text and the title. Every row links to the full capture (`archiveUrl`) and the raw original bytes (`rawArchiveUrl`).

# Actor input Schema

## `urls` (type: `array`):

Pages or sites to look up: example.com, https://example.com/pricing, *.example.com (all subdomains) or example.com/blog/* (everything under a path). Leave empty for a demo: one snapshot per year of apify.com.

## `mode` (type: `string`):

Snapshots: one row per capture of each URL. Archived URLs: one row per distinct URL the Wayback Machine has under a domain or path (with first capture date). Site history summary: one row per URL with first and last capture, months and years archived.

## `matchType` (type: `string`):

Auto: exact URL for snapshots and summary; whole domain (or path prefix if you give a path) for archived URLs. Prefix matches every URL starting with the input. Host matches one host. Domain matches the host and all subdomains.

## `collapse` (type: `string`):

Snapshots mode only. Content changes keeps a capture only when the page content (digest) changed from the previous one. Hour, day, month or year keeps the first capture in each period.

## `order` (type: `string`):

Snapshots mode only. Oldest first pages through all captures. Newest first returns the latest captures (up to 10,000 per URL).

## `dateFrom` (type: `string`):

YYYY, YYYY-MM or YYYY-MM-DD (UTC).

## `dateTo` (type: `string`):

YYYY, YYYY-MM or YYYY-MM-DD (UTC).

## `statusCodes` (type: `array`):

Keep only captures with these original HTTP status codes: 200, 3xx, 404. Prefix with ! to exclude (!404). Empty keeps all.

## `mimeTypes` (type: `array`):

Keep only these MIME types: text/html, application/pdf, image/\*. Prefix with ! to exclude. Empty keeps all.

## `includePageText` (type: `boolean`):

Snapshots mode only. For HTML captures with status 200, download the archived page and add its title, meta description and visible text. Charged as an extra page-text event per page with text.

## `maxTextChars` (type: `integer`):

Page text is cut after this many characters (textTruncated shows when).

## `maxResultsPerUrl` (type: `integer`):

Stop each URL or domain after this many snapshots or archived URLs.

## `maxItems` (type: `integer`):

Stop the run after this many paid rows. 0 means no limit (the max cost per run still applies).

## `startUrls` (type: `array`):

Same as URLs or domains, for runs started from another Actor or a request list.

## Actor input object example

```json
{
  "urls": [
    "apify.com/pricing"
  ],
  "mode": "snapshots",
  "matchType": "auto",
  "collapse": "none",
  "order": "oldest",
  "statusCodes": [],
  "mimeTypes": [],
  "includePageText": false,
  "maxTextChars": 20000,
  "maxResultsPerUrl": 1000,
  "maxItems": 0
}
```

# Actor output Schema

## `results` (type: `string`):

Snapshots, archived URLs or history summaries, plus free status rows for empty, failed or invalid inputs.

## `runSummary` (type: `string`):

JSON summary: rows saved per URL, CDX requests, rate limit pauses and stop reason.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "apify.com/pricing"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("joesimo/wayback-machine").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["apify.com/pricing"] }

# Run the Actor and wait for it to finish
run = client.actor("joesimo/wayback-machine").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "apify.com/pricing"
  ]
}' |
apify call joesimo/wayback-machine --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,joesimo/wayback-machine"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0IWfQYye8pMIeyUhY/builds/V4S5pGSctZc3SjdRJ/openapi.json
