Internet Archive Wayback Snapshots

Query the public Internet Archive Wayback CDX API for historical snapshots of a URL or domain. The Actor returns snapshot metadata, builds public replay URLs, and can optionally fetch readable text from the first selected archived pages.

No authentication or upstream Actor is used. The Wayback Machine may rate-limit requests or have gaps in its archive; a missing snapshot is not evidence that a page never existed.

What this Actor does

• Searches exact URLs, prefixes, hosts, or domains.
• Filters by date boundary, HTTP status, and MIME type.
• Uses CDX collapse modes such as digest, monthly, daily, or hourly.
• Maps CDX rows to stable JSON records with archive metadata.
• Optionally fetches and cleans archived HTML text for up to maxContentFetch records.
• Completes no-result searches successfully with an empty dataset and writes an OUTPUT summary for every run.
• Validates HTTP status, content type, response size, and CDX payload shape before mapping records.
• Retries temporary network, rate-limit, and upstream failures with bounded exponential backoff.

Input

{
  "url": "example.com",
  "matchType": "prefix",
  "dateFrom": "20200101",
  "dateTo": "20241231",
  "statusFilter": "200",
  "mimeFilter": "text/html",
  "collapseBy": "digest",
  "maxResults": 100,
  "includeContent": true,
  "maxContentFetch": 5,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 30
}

Field	Required	Description
url	Yes	URL or domain. Missing http:// or https:// is treated as HTTPS.
matchType	No	exact, prefix, host, or domain. Defaults to exact.
dateFrom / dateTo	No	YYYY, YYYYMM, or YYYYMMDD CDX boundaries.
statusFilter	No	Three-digit original HTTP status, such as 200.
mimeFilter	No	MIME type such as text/html or application/pdf.
collapseBy	No	none, digest, monthly, daily, or hourly. Defaults to digest.
maxResults	No	Integer from 1 to 10,000. Defaults to 500.
includeContent	No	Fetch readable archived text for selected snapshots. Defaults to false.
maxContentFetch	No	Integer from 0 to 500. Defaults to 10.
maxRequestRetries	No	Temporary-failure retries from 0 to 10. Defaults to 3.
requestTimeoutSecs	No	Per-request timeout from 5 to 120 seconds. Defaults to 30.

Unknown properties and invalid filter formats are rejected.

Output

{
  "recordType": "wayback-snapshot",
  "status": "success",
  "dataAvailable": true,
  "source": "web.archive.org",
  "provenance": "public_wayback_cdx_api",
  "sourceTransport": "fetch",
  "extractionMethod": "cdx_json_row_map",
  "originalUrl": "https://example.com/about",
  "archiveDate": "2020-01-15T12:34:56.000Z",
  "timestamp": "20200115123456",
  "archiveUrl": "https://web.archive.org/web/20200115123456id_/https://example.com/about",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "ABC123...",
  "contentLength": 18342,
  "archiveOffset": 123456,
  "archiveFilename": "WEB-20200115123456-example.warc.gz",
  "sourceUrl": "https://web.archive.org/cdx/search/cdx?...",
  "retrievedAt": "2026-08-18T12:00:00.000Z",
  "contentText": "Readable archived page text...",
  "contentFetched": true
}

contentText is present only when content fetching is enabled and an HTML replay succeeds. If an individual archive replay fails, the snapshot record keeps contentFetched: false and a safe contentError message. A valid search with no snapshots completes successfully with an empty dataset and dataAvailable: false in OUTPUT. A request or payload failure exits non-zero, leaves the dataset free of placeholder records, and records the safe failure summary in OUTPUT.

The OUTPUT key contains a summary:

{
  "recordType": "wayback-summary",
  "status": "success",
  "dataAvailable": true,
  "actor": "wayback-machine-search",
  "source": "web.archive.org",
  "provenance": "public_wayback_cdx_api",
  "query": "https://example.com",
  "matchType": "prefix",
  "dateFrom": "20200101",
  "dateTo": "20241231",
  "maxResults": 100,
  "includeContent": true,
  "maxContentFetch": 5,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 30,
  "itemsStored": 100,
  "contentFetchedCount": 5,
  "diagnosticCount": 0,
  "durationMs": 4200,
  "finishedAt": "2026-08-18T12:00:00.000Z"
}

Storage

• Dataset: normalized, deduplicated Wayback snapshot metadata and optional archived HTML text.
• Key-value store: OUTPUT contains query/filter context, counts, status, and duration.

Local verification

npm ci
npm test
npm run check
apify validate-schema
apify run --purge --input '{"url":"example.com","matchType":"prefix","maxResults":5,"includeContent":false}'

The implementation uses only public CDX and replay HTTP requests. It does not use credentials, cookies, browser fingerprinting, or CAPTCHA/access-control bypasses. Direct requests are appropriate for this public endpoint; proxy configuration is intentionally not exposed.

Cost and limits

Apify compute, dataset, and key-value-store charges follow your Apify plan. CDX queries with broad domain scopes and optional archived-content fetches take more time and bandwidth. Start with a narrow date range and small result limit.

Responsible use

Archived pages can contain copyrighted, personal, or sensitive information. Respect Internet Archive and original-site terms, use archived text only when you have a lawful purpose, and do not assume archived content is current or authoritative.

FAQ

Why does a snapshot have a different URL or status?

The record reports the original URL and the original response metadata captured by the Wayback CDX index. The replay page can behave differently today.

Does collapseBy: digest remove all duplicates?

It asks the CDX API to collapse adjacent snapshots with the same digest. It is not a general duplicate guarantee across every URL or time range.

Why is archived text missing?

Content fetching is opt-in, limited by maxContentFetch, and can fail when a replay is unavailable or non-HTML.