# Wayback Machine CDX Scraper (`devilscrapes/wayback-machine-cdx-scraper`) Actor

Query the Internet Archive Wayback Machine CDX Server API at scale: every URL a domain ever had captured, filtered by status code, MIME type, and date range, with correct resumeKey pagination and a stable validated schema. Keyless, no API key required.

- **URL**: https://apify.com/devilscrapes/wayback-machine-cdx-scraper.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<div align="center">
  <img src=".actor/icon.svg" width="160" alt="Devil Scrapes mark" />

## Wayback Machine CDX Scraper

**💰 $2.05 / 1 000 results**  ·  pay only for results  ·  no credit card to try

*We do the dirty work so your dataset stays clean.* 😈

Query the Internet Archive Wayback Machine CDX Server API at scale: every URL a domain ever had captured, filtered by status code, MIME type, and date range, with correct resumeKey pagination and a stable validated schema. Keyless, no API key required.

</div>

***

### 🎯 What this scrapes

The Internet Archive publishes a CDX Server API (<code>web.archive.org/cdx/search/cdx</code>) that indexes every snapshot it has ever taken of a URL, host, or domain — timestamp, original URL, MIME type, HTTP status, content digest, and length. This Actor drives that API for you: it builds the right <code>matchType</code>/<code>filter</code>/<code>collapse</code>/date-range query, walks <code>resumeKey</code> pagination across tens of thousands of rows without losing a page, and writes one clean row per capture. Point it at a domain to rebuild your full historical URL inventory (handy after a migration, when pages that used to rank 301 into oblivion), or at a single page to trace every version the Archive ever saw.

### 🔥 What we handle for you

- 🔁 **We retry with exponential backoff** on `408 / 429 / 503` and honour `Retry-After` — up to 5 attempts per page.
- 🧭 **We paginate correctly** — `resumeKey` chained page by page, stopping the instant a short page or your `maxResults` cap says we're done, so you never get truncated or duplicated rows.
- 🧱 **We fault-isolate bad pages** — one page that fails after retries gets skipped and logged; it never kills the whole run.
- 🧊 **We keep the dataset clean** — Pydantic-validated rows, fixed-position column mapping (never guessed from headers), ISO-8601 timestamps.
- 💰 **You pay only for results that land.** No data → no charge beyond the small `actor-start` warm-up fee.

### 💡 Use cases

- **Post-migration URL recovery** — rebuild the full list of URLs a domain ever had so you can 301 every one of them correctly instead of guessing.
- **OSINT / domain diligence** — reconstruct a site's publishing history: when pages appeared, what they served, when they changed.
- **Historical content audits** — filter by status code and MIME type to find every HTML page that was once live under a domain.
- **Change monitoring** — pull digests over a date range to see exactly when a page's content last changed.

### ⚙️ How to use it

1. Click **Try for free** at the top of the page.
2. Fill in the input form — most fields have sensible defaults.
3. Click **Start**. Output streams into the run's dataset.
4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or fetch via the API.

### 📥 Input

| Field | Type | Required | Default | Notes |
|---|---|:--:|---|---|
| `url` | `string` | **yes** | 'stripe.com' | Target URL, host, or domain to look up in the Wayback Machine CDX index, e.g. "stripe.com" or… |
| `matchType` | `string` | no | 'domain' | How the URL is matched: an exact URL, a URL prefix, an exact host, or the whole domain (every subdomain and path — the… |
| `dateFrom` | `string` | no | '—' | Start of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no lower… |
| `dateTo` | `string` | no | '—' | End of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no upper… |
| `filters` | `array` | no | \[] | Repeatable CDX filter expressions, e.g. "statuscode:200", "!statuscode:404", "mimetype:text/html". |
| `collapse` | `string` | no | '—' | CDX collapse expression to de-duplicate adjacent rows sharing a field, e.g. "urlkey" or "timestamp:8". Leave blank to… |
| `pageSize` | `integer` | no | 1000 | Rows requested per CDX page (the API's "limit" parameter). |
| `maxResults` | `integer` | no | 10000 | Hard cap on total rows fetched across all pages — protects spend and compute. |
| `proxyConfiguration` | `object` | no | {'useApifyProxy': False} | web.archive.org is keyless and not known to fingerprint clients at reasonable volumes — proxy is off by default. Switch… |

#### Example input

```json
{
  "url": "stripe.com",
  "matchType": "domain",
  "pageSize": 5,
  "maxResults": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

### 📤 Output

Every row is one dataset item.

| Field | Type | Notes |
|---|---|---|
| `urlkey` | `string` | SURT canonical key for the captured URL (CDX column 0). |
| `timestamp` | `string` | Raw 14-digit capture timestamp (CDX column 1). |
| `capture_date` | `string` | capture timestamp converted to ISO 8601 UTC. |
| `original_url` | `string` | The URL as originally captured (CDX column 2). |
| `mimetype` | `['string', 'null']` | Content MIME type at capture time (CDX column 3), null when the Archive recorded "-". |
| `statuscode` | `['string', 'null']` | HTTP status code at capture time (CDX column 4), null when the Archive recorded "-". |
| `digest` | `string` | Content digest / checksum for this capture (CDX column 5). |
| `length` | `['integer', 'null']` | Captured response length in bytes (CDX column 6), null when not numeric. |
| `archive_url` | `string` | Direct playback URL for this snapshot on web.archive.org. |
| `scraped_at` | `string` | When this row was recorded by the Actor, ISO 8601 UTC. |

#### Example output

```json
{
  "urlkey": "com,stripe)/",
  "timestamp": "20260815120133",
  "capture_date": "2026-08-15T12:01:33+00:00",
  "original_url": "https://stripe.com/",
  "mimetype": "text/html",
  "statuscode": "200",
  "digest": "3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ",
  "length": 18452,
  "archive_url": "https://web.archive.org/web/20260815120133/https://stripe.com/",
  "scraped_at": "2026-09-01T12:00:00+00:00"
}
```

### 💰 Pricing

Pay-Per-Event — you pay only when these events fire:

| Event | USD | What it is |
|---|---:|---|
| `actor-start` | $0.05 | One-off warm-up charge per run |
| `snapshot-scraped` | $0.002 | PPE event |

Example: 1 000 results at the rates above ≈ **$2.05**. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

### 🚧 Limitations

This Actor covers the CDX Server API's index only — it does not fetch archived page content, use the plain-text `output=text` mode, expose a custom field list, or touch the Wayback Machine's save/write endpoints. `maxResults` is our own spend/compute safety cap, not an Internet Archive limit.

### ❓ FAQ

**Does this fetch the archived page content?**

No — this Actor returns the CDX index (capture metadata: timestamp, status, MIME type, digest, length), not rendered HTML. Each row's `archive_url` is a direct link you can fetch yourself or feed into a follow-up scrape.

**What does "domain" match type actually match?**

`domain` returns every URL under the target domain across all subdomains and paths — the flagship "recover my whole site's history" use case. Use `host` to stay on one hostname, `prefix` for a URL prefix, or `exact` for one exact URL.

**Why are some `statuscode` or `length` values null?**

The Internet Archive records `-` for some captures (redirect-only or malformed responses). The Actor converts those to `null` rather than treating them as errors.

**What happens if my query matches nothing?**

The run finishes successfully with zero rows and a status message describing exactly what was searched — a narrow filter with no matches is not a failure.

### 💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an
issue on the Actor's **Issues** tab on Apify Console — we ship
fixes weekly and we read every report.

***

<div align="center">

Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of
opinionated public-data Actors. Honest pricing, real engineering, zero fine print.

</div>

# Actor input Schema

## `url` (type: `string`):

Target URL, host, or domain to look up in the Wayback Machine CDX index, e.g. "stripe.com" or "https://stripe.com/pricing".

## `matchType` (type: `string`):

How the URL is matched: an exact URL, a URL prefix, an exact host, or the whole domain (every subdomain and path — the "recover my whole site's history" default).

## `dateFrom` (type: `string`):

Start of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no lower bound.

## `dateTo` (type: `string`):

End of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no upper bound.

## `filters` (type: `array`):

Repeatable CDX filter expressions, e.g. "statuscode:200", "!statuscode:404", "mimetype:text/html".

## `collapse` (type: `string`):

CDX collapse expression to de-duplicate adjacent rows sharing a field, e.g. "urlkey" or "timestamp:8". Leave blank to disable.

## `pageSize` (type: `integer`):

Rows requested per CDX page (the API's "limit" parameter).

## `maxResults` (type: `integer`):

Hard cap on total rows fetched across all pages — protects spend and compute.

## `proxyConfiguration` (type: `object`):

web.archive.org is keyless and not known to fingerprint clients at reasonable volumes — proxy is off by default. Switch it on if you hit IP-range throttling on very large pulls.

## Actor input object example

```json
{
  "url": "stripe.com",
  "matchType": "domain",
  "filters": [],
  "pageSize": 1000,
  "maxResults": 10000,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

All dataset items as JSON.

## `datasetItemsCsv` (type: `string`):

Same data exported to CSV.

## `datasetView` (type: `string`):

Open the run dataset in the Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "stripe.com",
    "matchType": "domain",
    "filters": [],
    "pageSize": 1000,
    "maxResults": 10000,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/wayback-machine-cdx-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "stripe.com",
    "matchType": "domain",
    "filters": [],
    "pageSize": 1000,
    "maxResults": 10000,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/wayback-machine-cdx-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "stripe.com",
  "matchType": "domain",
  "filters": [],
  "pageSize": 1000,
  "maxResults": 10000,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call devilscrapes/wayback-machine-cdx-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devilscrapes/wayback-machine-cdx-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8aqBd2My0dokd81zM/builds/JhpXcMBJMwNSYCSBr/openapi.json