Wayback Machine Snapshot History Scraper

Export Internet Archive CDX snapshot histories for domains, hosts, URL prefixes, or exact URLs.

Get capture timestamps, original URLs, HTTP statuses, MIME types, content digests, byte lengths, and ready-to-open replay URLs in a structured Apify dataset.

The Actor uses the public Wayback Machine CDX API directly. It needs no browser, login, API token, cookie, or proxy.

What does this Wayback Machine scraper do?

Wayback Machine Snapshot History Scraper turns Internet Archive CDX metadata into analysis-ready records.

Use it to:

• inventory archived URLs before a site migration;
• find historical versions of a page;
• build redirect and content-recovery maps;
• compare digests to identify changed captures;
• document when public web content appeared or changed;
• schedule recurring archival checks;
• export replay links for review by researchers.

This Actor exports capture metadata. It does not download or parse the archived page body.

Who is it for?

SEO and migration teams can recover old URL inventories and plan redirects.

Site owners can locate recoverable versions of lost pages.

OSINT and legal researchers can create dated, source-linked capture timelines.

Analysts and journalists can monitor historical changes to public pages.

Developers can feed normalized archive metadata into databases, spreadsheets, or change-detection pipelines.

Why use this Actor?

• Multiple input scopes: domain, host, URL prefix, or exact URL.
• Date, HTTP status, and MIME type filters are applied by CDX.
• Optional digest collapse removes unchanged duplicate captures.
• Every row includes a direct replay URL.
• Pagination respects a run-wide maximum item count.
• Transient timeouts, 429 responses, and 5xx errors use bounded backoff.
• Partial rows already saved remain available if a later request fails.
• Typed dataset views work with Apify integrations and API clients.
• Direct public HTTP keeps memory and infrastructure cost low.

What data can I extract?

Field	Meaning
sourceQuery	The input target that produced the record
captureTimestamp	CDX UTC timestamp in YYYYMMDDhhmmss format
captureDate	The same capture time as ISO 8601
originalUrl	URL recorded by Internet Archive
statusCode	Archived HTTP response status, when available
mimeType	Archived response MIME type, when available
digest	Internet Archive content digest for change analysis
byteLength	CDX compressed record length, when available
replayUrl	Direct Wayback replay URL using id_ mode
cdxApiUrl	Exact CDX API page request used by the Actor
fetchedAt	Time when the metadata page was fetched

Fields unavailable in an upstream CDX row are returned as null.

How to scrape Wayback Machine snapshot history

1. Open the Actor input form.
2. Add one or more domains, URLs, or patterns to Targets.
3. Select the match type that describes the intended scope.
4. Optionally set from, to, status, MIME, and digest-collapse filters.
5. Choose a practical maxItems limit.
6. Click Start.
7. Open the Snapshot history dataset view.
8. Export records as JSON, CSV, Excel, XML, or RSS.

Start with a narrow exact URL or date range before requesting a large domain.

Input parameters

targets

Required list of 1–100 domains, URLs, or URL patterns.

Examples:

• nasa.gov
• https://www.whitehouse.gov/
• https://docs.python.org/3/

matchType

• exact: only the supplied URL;
• prefix: URLs beginning with the supplied prefix;
• host: one hostname;
• domain: a domain and its subdomains.

The default is domain.

from and to

Optional UTC capture boundaries using 4–14 digits.

Examples include 2020, 20200101, and 20200101120000.

statusCodes

Optional exact codes such as 200 and 301, or classes such as 2...

mimeTypes

Optional MIME filters such as text/html, application/pdf, or image/*.

collapseByDigest

When enabled, CDX collapses repeated records with the same content digest.

maxItems

Run-wide maximum across every target, from 1 to 100,000.

pageSize

CDX page size from 1 to 2,000. The default is 500.

maxRetries

Retry count for transient failures, from 0 to 8. The default is 4.

Example input

{
  "targets": ["https://www.whitehouse.gov/"],
  "matchType": "exact",
  "from": "20200101",
  "to": "20201231",
  "statusCodes": ["200"],
  "mimeTypes": ["text/html"],
  "collapseByDigest": true,
  "maxItems": 250
}

Example output

{
  "sourceQuery": "https://www.whitehouse.gov/",
  "captureTimestamp": "20200101034743",
  "captureDate": "2020-01-01T03:47:43Z",
  "originalUrl": "https://www.whitehouse.gov/",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "5U4R6QJEOVRLUWTNMLD6NNM6HL64OKLI",
  "byteLength": 16014,
  "replayUrl": "https://web.archive.org/web/20200101034743id_/https://www.whitehouse.gov/",
  "cdxApiUrl": "https://web.archive.org/cdx/search/cdx?...",
  "fetchedAt": "2026-07-27T03:15:00.000Z"
}

This example reflects real CDX output. fetchedAt varies by run.

How much does it cost to export Wayback Machine captures?

The Actor uses pay-per-event pricing: one Run started event per run and one Item extracted event per saved capture.

The BRONZE price is $0.004 per run plus $0.000035714 per capture. Higher subscription tiers receive progressively lower per-item prices as shown in the Apify pricing tab.

At the BRONZE item price:

• 10 captures cost about $0.00436 including the start event;
• 100 captures cost about $0.00757 including the start event;
• 1,000 captures cost about $0.03971 including the start event.

You are never charged an item event for a rejected, duplicate, empty, or failed row. Apify displays the active tier and final run charge before billing.

SEO migration and redirect-map workflow

1. Query the old domain with matchType: domain.
2. Restrict dates to the period before migration.
3. Select successful HTML captures.
4. Enable digest collapse to reduce unchanged duplicates.
5. Export originalUrl, captureDate, digest, and replayUrl.
6. Group by original URL.
7. Review the newest replay for each old URL.
8. Map recovered URLs to current destinations.

For large sites, split work by hostname, path prefix, or year.

Historical change timeline workflow

Schedule the Actor with an exact URL and digest collapse enabled.

Store rows in Google Sheets, Airtable, a webhook receiver, or your data warehouse.

A new digest can indicate changed archived content. Digest equality is a useful signal, but it is not a semantic comparison of rendered pages.

Export and integrations

The default dataset supports:

• JSON, JSONL, CSV, Excel, XML, and RSS exports;
• Google Sheets and Zapier integrations;
• Make workflows;
• webhooks after a successful run;
• JavaScript and Python API clients;
• direct dataset API access;
• Apify schedules for repeat monitoring.

Use sourceQuery to separate records when one run contains several targets.

Run with the Apify API using cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~wayback-machine-snapshot-history-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"targets":["nasa.gov"],"matchType":"domain","statusCodes":["200"],"mimeTypes":["text/html"],"collapseByDigest":true,"maxItems":100}'

The response contains the run ID and default dataset ID.

Run with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/wayback-machine-snapshot-history-scraper').call({
  targets: ['https://docs.python.org/3/'],
  matchType: 'prefix',
  statusCodes: ['2..'],
  mimeTypes: ['text/html'],
  collapseByDigest: true,
  maxItems: 1000,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Run with Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/wayback-machine-snapshot-history-scraper').call(run_input={
    'targets': ['https://www.whitehouse.gov/'],
    'matchType': 'exact',
    'from': '20200101',
    'to': '20201231',
    'statusCodes': ['200'],
    'mimeTypes': ['text/html'],
    'maxItems': 250,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/wayback-machine-snapshot-history-scraper"

Claude Desktop, Cursor, and VS Code setup

Claude Desktop, Cursor, and VS Code can use this MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/wayback-machine-snapshot-history-scraper"
    }
  }
}

Example prompts:

• “Export successful NASA HTML snapshots captured between 2018 and 2020.”
• “Find digest-distinct versions of the White House homepage during 2020.”
• “Build a historical URL inventory below the Python 3 docs prefix.”

Limits and responsible scaling

Internet Archive is a public service and may throttle broad or frequent queries.

Keep pageSize and concurrency conservative. This Actor processes targets sequentially.

A domain query may represent millions of records. Use date ranges, path prefixes, filters, and maxItems to make work bounded.

The Actor retries temporary failures but does not retry malformed input or deterministic 4xx responses.

No automatic proxy fallback is used.

Data freshness and interpretation

CDX describes captures indexed by Internet Archive; it is not a live crawl of the target site.

A missing result does not prove that a page never existed.

Indexing delays, exclusions, takedowns, robots policies, and archive availability can affect results.

A replay URL may be unavailable even when a CDX record exists.

byteLength is CDX metadata and should not be treated as the uncompressed page size.

Legal and responsible use

Use archived public-web metadata for lawful purposes.

Respect Internet Archive terms, source-site rights, privacy obligations, and applicable retention or evidence rules.

Do not use historical pages to expose sensitive personal data, evade access controls, or misrepresent archived content as current.

For legal or evidentiary work, preserve run input, timestamps, dataset exports, and independent chain-of-custody records. This Actor does not certify authenticity.

Troubleshooting

Why did the dataset contain no records?

Check the target spelling, matchType, date range, status filters, and MIME filters. Try an exact known URL with fewer filters.

Why did the run report HTTP 429 or a timeout?

Reduce pageSize, narrow the date or URL scope, and retry later. The Actor already applies bounded exponential backoff.

Why are there many similar captures?

Enable collapseByDigest. Separate captures can still share similar content while having different digests.

Why are some metadata values null?

Older or unusual CDX records may omit a status, MIME type, digest, or length.

Why does a replay link fail?

Archive replay availability can differ from CDX indexing. Try the standard Wayback calendar for the original URL.

FAQ

Does it download archived page content?

No. It exports snapshot metadata and replay links only.

Does it require an Internet Archive account?

No. It uses the anonymous public CDX endpoint.

Does it use an Apify proxy?

No. Requests are direct, so users do not incur proxy transfer for this Actor.

Can it find every URL ever archived for a domain?

It can query domain scope, but complete histories can be very large and upstream availability is not guaranteed. Use bounded filters and limits.

Can I monitor several targets in one run?

Yes. Add up to 100 targets. maxItems applies across the whole run.

How do I detect changes?

Enable digest collapse or compare digest values over captureDate. For semantic differences, fetch authorized replay pages in a downstream process.

Related automation-lab Actors

This Actor is intentionally standalone because no active automation-lab Actor currently provides the same Wayback CDX snapshot-history workflow.

Combine its dataset with your existing URL audit, redirect mapping, spreadsheet, webhook, or data-warehouse automation.

Only automation-lab Actors will be linked here when a genuinely complementary published product is available.