# Wayback Machine Search Scraper (`searchapi/wayback-machine-search`) Actor

Export Internet Archive Wayback Machine snapshot history with replay URLs, timestamps, status, MIME, digest, and size filters.

- **URL**: https://apify.com/searchapi/wayback-machine-search.md
- **Developed by:** [Search API](https://apify.com/searchapi) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.99 / 1,000 search results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Internet Archive Wayback Snapshots

Query the public Internet Archive Wayback CDX API for historical snapshots of a URL or domain. The Actor returns snapshot metadata, builds public replay URLs, and can optionally fetch readable text from the first selected archived pages.

No authentication or upstream Actor is used. The Wayback Machine may rate-limit requests or have gaps in its archive; a missing snapshot is not evidence that a page never existed.

### What this Actor does

- Searches exact URLs, prefixes, hosts, or domains.
- Filters by date boundary, HTTP status, and MIME type.
- Uses CDX collapse modes such as digest, monthly, daily, or hourly.
- Maps CDX rows to stable JSON records with archive metadata.
- Optionally fetches and cleans archived HTML text for up to `maxContentFetch` records.
- Completes no-result searches successfully with an empty dataset and writes an `OUTPUT` summary for every run.
- Validates HTTP status, content type, response size, and CDX payload shape before mapping records.
- Retries temporary network, rate-limit, and upstream failures with bounded exponential backoff.

### Input

```json
{
  "url": "example.com",
  "matchType": "prefix",
  "dateFrom": "20200101",
  "dateTo": "20241231",
  "statusFilter": "200",
  "mimeFilter": "text/html",
  "collapseBy": "digest",
  "maxResults": 100,
  "includeContent": true,
  "maxContentFetch": 5,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 30
}
```

| Field | Required | Description |
|---|---:|---|
| `url` | Yes | URL or domain. Missing `http://` or `https://` is treated as HTTPS. |
| `matchType` | No | `exact`, `prefix`, `host`, or `domain`. Defaults to `exact`. |
| `dateFrom` / `dateTo` | No | `YYYY`, `YYYYMM`, or `YYYYMMDD` CDX boundaries. |
| `statusFilter` | No | Three-digit original HTTP status, such as `200`. |
| `mimeFilter` | No | MIME type such as `text/html` or `application/pdf`. |
| `collapseBy` | No | `none`, `digest`, `monthly`, `daily`, or `hourly`. Defaults to `digest`. |
| `maxResults` | No | Integer from 1 to 10,000. Defaults to `500`. |
| `includeContent` | No | Fetch readable archived text for selected snapshots. Defaults to `false`. |
| `maxContentFetch` | No | Integer from 0 to 500. Defaults to `10`. |
| `maxRequestRetries` | No | Temporary-failure retries from 0 to 10. Defaults to `3`. |
| `requestTimeoutSecs` | No | Per-request timeout from 5 to 120 seconds. Defaults to `30`. |

Unknown properties and invalid filter formats are rejected.

### Output

```json
{
  "recordType": "wayback-snapshot",
  "status": "success",
  "dataAvailable": true,
  "source": "web.archive.org",
  "provenance": "public_wayback_cdx_api",
  "sourceTransport": "fetch",
  "extractionMethod": "cdx_json_row_map",
  "originalUrl": "https://example.com/about",
  "archiveDate": "2020-01-15T12:34:56.000Z",
  "timestamp": "20200115123456",
  "archiveUrl": "https://web.archive.org/web/20200115123456id_/https://example.com/about",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "ABC123...",
  "contentLength": 18342,
  "archiveOffset": 123456,
  "archiveFilename": "WEB-20200115123456-example.warc.gz",
  "sourceUrl": "https://web.archive.org/cdx/search/cdx?...",
  "retrievedAt": "2026-08-18T12:00:00.000Z",
  "contentText": "Readable archived page text...",
  "contentFetched": true
}
```

`contentText` is present only when content fetching is enabled and an HTML replay succeeds. If an individual archive replay fails, the snapshot record keeps `contentFetched: false` and a safe `contentError` message. A valid search with no snapshots completes successfully with an empty dataset and `dataAvailable: false` in `OUTPUT`. A request or payload failure exits non-zero, leaves the dataset free of placeholder records, and records the safe failure summary in `OUTPUT`.

The `OUTPUT` key contains a summary:

```json
{
  "recordType": "wayback-summary",
  "status": "success",
  "dataAvailable": true,
  "actor": "wayback-machine-search",
  "source": "web.archive.org",
  "provenance": "public_wayback_cdx_api",
  "query": "https://example.com",
  "matchType": "prefix",
  "dateFrom": "20200101",
  "dateTo": "20241231",
  "maxResults": 100,
  "includeContent": true,
  "maxContentFetch": 5,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 30,
  "itemsStored": 100,
  "contentFetchedCount": 5,
  "diagnosticCount": 0,
  "durationMs": 4200,
  "finishedAt": "2026-08-18T12:00:00.000Z"
}
```

### Storage

- Dataset: normalized, deduplicated Wayback snapshot metadata and optional archived HTML text.
- Key-value store: `OUTPUT` contains query/filter context, counts, status, and duration.

### Local verification

```bash
npm ci
npm test
npm run check
apify validate-schema
apify run --purge --input '{"url":"example.com","matchType":"prefix","maxResults":5,"includeContent":false}'
```

The implementation uses only public CDX and replay HTTP requests. It does not use credentials, cookies, browser fingerprinting, or CAPTCHA/access-control bypasses. Direct requests are appropriate for this public endpoint; proxy configuration is intentionally not exposed.

### Cost and limits

Apify compute, dataset, and key-value-store charges follow your Apify plan. CDX queries with broad domain scopes and optional archived-content fetches take more time and bandwidth. Start with a narrow date range and small result limit.

### Responsible use

Archived pages can contain copyrighted, personal, or sensitive information. Respect Internet Archive and original-site terms, use archived text only when you have a lawful purpose, and do not assume archived content is current or authoritative.

### FAQ

#### Why does a snapshot have a different URL or status?

The record reports the original URL and the original response metadata captured by the Wayback CDX index. The replay page can behave differently today.

#### Does `collapseBy: digest` remove all duplicates?

It asks the CDX API to collapse adjacent snapshots with the same digest. It is not a general duplicate guarantee across every URL or time range.

#### Why is archived text missing?

Content fetching is opt-in, limited by `maxContentFetch`, and can fail when a replay is unavailable or non-HTML.

# Actor input Schema

## `url` (type: `string`):

URL or domain to search. A missing scheme is treated as HTTPS.

## `matchType` (type: `string`):

CDX URL matching scope.

## `dateFrom` (type: `string`):

Earliest snapshot boundary: YYYY, YYYYMM, or YYYYMMDD.

## `dateTo` (type: `string`):

Latest snapshot boundary: YYYY, YYYYMM, or YYYYMMDD.

## `statusFilter` (type: `string`):

Optional three-digit HTTP status code.

## `mimeFilter` (type: `string`):

Optional MIME type filter such as text/html or application/pdf.

## `collapseBy` (type: `string`):

CDX adjacent-snapshot deduplication mode.

## `maxResults` (type: `integer`):

Maximum snapshot records to return.

## `includeContent` (type: `boolean`):

Fetch and clean archived page text for the first maxContentFetch records.

## `maxContentFetch` (type: `integer`):

Maximum archived pages to fetch when includeContent is true.

## `maxRequestRetries` (type: `integer`):

Retries for temporary network failures, timeouts, HTTP 429, and HTTP 5xx responses.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each CDX or archived-page HTTP request.

## Actor input object example

```json
{
  "url": "example.com",
  "matchType": "exact",
  "collapseBy": "digest",
  "maxResults": 500,
  "includeContent": false,
  "maxContentFetch": 10,
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `dataset` (type: `string`):

Public CDX snapshot records and diagnostics.

## `output` (type: `string`):

Summary stored in the OUTPUT key-value-store record.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("searchapi/wayback-machine-search").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("searchapi/wayback-machine-search").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call searchapi/wayback-machine-search --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,searchapi/wayback-machine-search"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TG3wdfNZz13xsPWOe/builds/RK2TvvRjd9y3UhQHe/openapi.json