# Wayback Machine URL Extractor – Archived URLs, Snapshots & Diff (`gazidev/wayback-url-extractor`) Actor

List every URL the Internet Archive Wayback Machine has saved for a domain or URL prefix: first and last capture, capture count, status, MIME type and snapshot links. Date, status and MIME filters. Optional archived page text as Markdown and a content diff between two dates.

- **URL**: https://apify.com/gazidev/wayback-url-extractor.md
- **Developed by:** [Cemal Atakli](https://apify.com/gazidev) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 archived urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Machine URL Extractor – Archived URLs, Snapshots & Diff

List **every URL the Internet Archive's Wayback Machine has saved** for a domain, subdomain or URL prefix. You get one clean row per unique URL with the **first and last capture date**, **capture count**, number of **content versions**, last HTTP status, MIME type and direct **snapshot links**. Optionally the Actor downloads the **archived page as Markdown or text**, or **compares a page between two dates** (similarity, lines added and removed, unified diff).

It uses the official Wayback CDX API, with no browser and no proxy, so it is fast and costs **$0.20 per 1,000 URLs**.

### What it does

- **All archived URLs of a site.** Give `example.com` (domain + subdomains), `example.com/blog/` (URL prefix) or an exact page URL. The CDX API results are de-duplicated by `urlkey`, so `http://`, `https://`, `www.` and `:80` variants of the same page become **one row**.
- **Capture statistics per URL:** `firstCapture`, `lastCapture`, `captureCount`, `uniqueVersions` (distinct content digests), `lastOkCapture` (last HTTP 200), the latest status code, MIME type and size.
- **Snapshot links:** `latestSnapshotUrl` (normal Wayback view), `latestRawSnapshotUrl` (`id_` raw archived bytes, no toolbar) and `firstSnapshotUrl`.
- **Filters on the Internet Archive server**, so nothing filtered is downloaded or charged:
  - date range (`from` / `to`: `2015`, `2019-06-30`, `2 years`),
  - HTTP status (`200`, `3xx`, `404`),
  - MIME type (`text/html`, `application/pdf`, `image/*`),
  - URL regex include and exclude,
  - skip query-string URLs.
- **Clean by default.** Tracking-parameter variants (`utm_*`, `fbclid`, `gclid`, `__hstc`, session IDs) and malformed URLs harvested from broken HTML (`"><span>`, `%22`, glued URLs) are dropped for free.
- **Snapshot content (optional).** The latest (or earliest) successful capture is downloaded in raw `id_` mode and converted to clean **Markdown**, plain text or raw HTML. This is charged per snapshot.
- **Diff between two dates (optional).** For each URL, the HTTP 200 capture closest to each date is downloaded, converted to Markdown and compared. You get `changed`, `similarity`, `linesAdded`, `linesRemoved`, `addedLines`, `removedLines` and a `unifiedDiff`.
- **Polite and robust.** One CDX request at a time with at least 1 s between calls, automatic slow-down and exponential backoff on HTTP 429/5xx (honouring `Retry-After`), an honest User-Agent with a contact address, and a safe stop if the archive keeps rate-limiting. Large result sets are paged with `resumeKey`. Very busy URLs (home pages with 100,000+ captures) are counted up to `maxCapturesPerUrl` and then skipped ahead, so the run never stalls on one URL.

### Use cases

- **SEO and site migrations:** find every old URL of a domain to build a 301 redirect map, recover lost pages and backlinks, and audit legacy URL structures.
- **Expired domain research:** see what a domain hosted, when, and how often it was crawled before you buy it.
- **Content recovery:** get deleted blog posts or docs back as Markdown from their last good capture.
- **Competitive intelligence:** track how a competitor's pricing, terms or landing pages changed between two dates.
- **Security and bug bounty recon:** list historical endpoints, parameters, `.js`, `.json`, `.pdf` and backup files (like `waybackurls` and `gau`, as a hosted API).
- **Research, journalism and compliance:** prove what a page said on a given date, with the archive link.
- **AI and RAG:** feed archived page text into LLMs for historical analysis.

### Input example

```json
{
  "urls": ["example.com", "example.org/blog/", "https://example.net/pricing"],
  "matchType": "auto",
  "maxUrls": 5000,
  "from": "2018",
  "to": "2024-12-31",
  "statusCodes": ["200"],
  "mimeTypes": ["text/html"],
  "excludeUrlRegex": ".*\\.(css|js|png|jpg|gif|svg|woff2?)",
  "captureStats": true,
  "fetchContent": false,
  "compareFrom": "",
  "compareTo": ""
}
```

The default input lists 200 archived URLs of `crawlee.dev` in about 3 seconds.

| Field | Default | Notes |
|---|---|---|
| `urls` | `crawlee.dev` | Domains, `*.domain`, URL prefixes (`/` or `*` at the end) or exact URLs |
| `matchType` | `auto` | `domain` / `host` / `prefix` / `exact` force one type for all entries |
| `maxUrls` / `maxUrlsPerInput` | 200 / 0 | 0 = no limit |
| `from` / `to` | – | `2015`, `2015-06`, `2015-06-30`, or relative `6 months` |
| `statusCodes` / `mimeTypes` | – | e.g. `200`, `3xx` / `text/html`, `image/*` |
| `urlRegex` / `excludeUrlRegex` | – | Java regex that must match the whole original URL |
| `skipQueryStrings` | false | Drop every URL with `?` |
| `skipTrackingUrls` | true | Drop `utm_*`, `fbclid`, `gclid`, session-ID variants only |
| `skipMalformedUrls` | true | Drop broken-HTML junk URLs |
| `captureStats` | true | Off: one CDX row per URL (faster on huge domains, only the first capture is known) |
| `maxCapturesPerUrl` | 5000 | Counting cap for very busy URLs |
| `fetchContent`, `contentFormat`, `contentSnapshot` | false, markdown, latest | Archived page text |
| `compareFrom` / `compareTo` | – | Set both to get a diff per URL |

### Output example

One dataset row per unique archived URL. The **Archived URLs**, **Snapshot content**, **Diff between two dates** and **Errors** views are in the Output tab.

```json
{
  "type": "url",
  "url": "https://apify.com/pricing",
  "host": "apify.com",
  "path": "/pricing",
  "extension": null,
  "urlkey": "com,apify)/pricing",
  "firstCapture": "2017-10-27T07:20:08Z",
  "lastCapture": "2026-10-03T03:43:41Z",
  "captureCount": 181,
  "captureCountCapped": false,
  "uniqueVersions": 176,
  "lastOkCapture": "2026-10-03T03:43:41Z",
  "mimeType": "text/html",
  "statusCode": 200,
  "lengthBytes": 55928,
  "firstSnapshotUrl": "https://web.archive.org/web/20171027072008/https://www.apify.com/pricing",
  "latestSnapshotUrl": "https://web.archive.org/web/20261003034341/https://apify.com/pricing",
  "latestRawSnapshotUrl": "https://web.archive.org/web/20261003034341id_/https://apify.com/pricing",
  "input": "https://apify.com/pricing",
  "matchType": "exact",
  "scrapedAt": "2026-10-03T09:46:40Z"
}
```

- With `fetchContent`, a row also has `contentStatus` (`ok` / `failed` / `skipped`), `contentSnapshotDate`, `contentSnapshotUrl`, `title`, `contentWordCount`, `content`, `contentTruncated` and `contentError`.
- With `compareFrom` and `compareTo`, a row also has `compareFromSnapshot`, `compareToSnapshot` (the actual capture dates used), `sameCapture`, `changed`, `similarity` (0–1), `linesAdded`, `linesRemoved`, `addedLines`, `removedLines`, `unifiedDiff`, `titleBefore`/`titleAfter`, `diffStatus` and `diffError`.
- Inputs with no captures, invalid inputs and refused queries become rows with `"type": "error"`, which are free.
- The key-value store record `OUTPUT` holds the run summary: per input, the URLs saved, captures scanned, CDX requests, malformed URLs skipped and why the run stopped.

See `SAMPLE_OUTPUT.json` for full rows with content and diff.

### Pricing

Pay per event. You pay only for URLs saved and snapshots downloaded.

| Event | Price |
|---|---|
| Archived URL saved | **$0.0002** ($0.20 / 1,000) |
| Snapshot fetched (content or diff, optional) | $0.001 ($1 / 1,000) |
| Actor start | $0.00005 |

A diff uses up to 2 snapshots per URL. If both dates resolve to the same capture, it is downloaded and charged only once. Non-text captures (images, PDFs) and failed downloads are not charged.

Compared with other Store Actors (public Store prices, 2026-10-03):

| Actor | Price per 1,000 URLs |
|---|---|
| **Wayback Machine URL Extractor (this Actor)** | **$0.20** |
| andok (Wayback) | $1.00 |
| ryanclinton/wayback-machine-search | $3.50 |
| logiover (Wayback) | $5.00 |

Set **Maximum cost per run** on the run options to cap spending. The Actor stops cleanly when the limit is reached and never downloads a snapshot it cannot charge for.

### FAQ

**Why do I get fewer URLs than tools like `waybackurls`?**
By default, rows are unique pages (`urlkey`). Tracking-parameter variants and malformed URLs are dropped. Set `skipTrackingUrls` and `skipMalformedUrls` to `false` to get the raw list.

**Some big sites return an error.**
For a few very large sites (for example nytimes.com and reddit.com), the Internet Archive refuses anonymous domain and prefix listings ("This type of CDX query requires authorization"). The Actor reports this as an error row, free of charge. Exact page URLs on those sites still work, including content and diff.

**What does `captureCountCapped` mean?**
Home pages can have hundreds of thousands of captures. After `maxCapturesPerUrl` captures (default 5,000), counting stops for that URL, the newest capture is looked up directly, and the scan moves on. Raise the limit for exact counts.

**Do filters change the statistics?**
Yes. `firstCapture`, `lastCapture` and `captureCount` count only the captures that match your date, status and MIME filters. For example, with `statusCodes: ["200"]`, `captureCount` is the number of successful captures.

**How is "latest snapshot" chosen?**
`latestSnapshotUrl` and the content download use the newest **HTTP 200** capture (`lastOkCapture`), so you do not land on an archived 404 or redirect. `lastCapture` and `statusCode` describe the newest capture of any status.

**How fast is it?**
Listing runs at about 10,000 captures per CDX request, at most one request per second. With `captureStats` off, one CDX row is one URL, so huge domains list quickly. Snapshot downloads run 2 at a time with at least 0.5 s between requests, out of respect for the Internet Archive's free service.

**Is it legal?**
The Actor uses the Internet Archive's public CDX and Wayback APIs at a polite rate, identifies itself, and does not bypass any exclusion. The archived content belongs to its original owners, so check copyright before you republish it. If you use the Internet Archive heavily, please consider [donating to it](https://archive.org/donate).

### Use with AI agents / Apify MCP

- **Apify MCP server:** add `gazidev/wayback-url-extractor` to your MCP client (Claude Desktop, Cursor, VS Code) via `https://mcp.apify.com?actors=gazidev/wayback-url-extractor`. An agent can call it with `{"urls":["https://competitor.com/pricing"],"compareFrom":"1 year","compareTo":"1 day"}` and get back what changed on the page.
- **API:** `POST https://api.apify.com/v2/acts/gazidev~wayback-url-extractor/run-sync-get-dataset-items?token=...` with the input JSON returns the rows directly.
- **Scheduled monitoring:** a weekly Task with `compareFrom: "7 days"` and `compareTo: "1 day"` on key competitor pages can feed the Slack, Google Sheets or webhook integration.

### More from the website toolkit

- [Sitemap Extractor](https://apify.com/gazidev/sitemap-url-extractor): every URL a site lists in its sitemap.xml today. Diff it against this Actor's output to find removed pages.
- [Website to Markdown](https://apify.com/gazidev/website-to-markdown): any live URL, sitemap or whole site to clean Markdown with RAG chunks and llms.txt.
- [SEO Audit Crawler](https://apify.com/gazidev/seo-audit-crawler): broken links, meta tags and redirects.
- [RSS Feed Reader](https://apify.com/gazidev/rss-feed-reader): any RSS, Atom or JSON Feed to JSON, with only-new mode.
- [Tech Stack Detector](https://apify.com/gazidev/tech-stack-detector): bulk Wappalyzer alternative.
- [Domain Checker](https://apify.com/gazidev/domain-intel): WHOIS/RDAP, DNS, SSL and email security in bulk.
- [Website Contact Finder](https://apify.com/gazidev/website-contact-finder): emails, phones and social links from websites.

Not affiliated with the Internet Archive. Wayback Machine is a trademark of the Internet Archive.

# Actor input Schema

## `urls` (type: `array`):

Any mix of: a domain (`example.com`: every archived URL of the domain and its subdomains), `*.example.com`, a URL prefix (`example.com/blog/` or `example.com/docs/*`: everything under that path) or an exact page URL (`https://example.com/pricing`: that page only, useful for the diff). Change how entries are read with `Match type`.

## `matchType` (type: `string`):

`auto`: bare domain → domain incl. subdomains, URL ending in `/` or `*` → prefix, other URLs → exact page. Or force one type for all entries: `domain` (domain + all subdomains), `host` (this host only, no subdomains), `prefix` (all URLs starting with the entry) or `exact`.

## `maxUrls` (type: `integer`):

Stop after saving this many unique archived URLs in total (0 = no limit). You pay only for URLs saved. Large domains can have millions of archived URLs.

## `maxUrlsPerInput` (type: `integer`):

Limit per entry when you give several (0 = no per-entry limit).

## `from` (type: `string`):

Only captures on or after this date. `2015`, `2015-06`, `2015-06-30` or relative (`1 year`, `6 months`, `30 days`).

## `to` (type: `string`):

Only captures on or before this date (inclusive: `2019` means until 2019-12-31). Same formats as above.

## `statusCodes` (type: `array`):

Keep only captures with these original HTTP statuses, e.g. `200`, or `3xx` for all redirects. Empty = all statuses (incl. 404s and redirects that the crawler recorded).

## `mimeTypes` (type: `array`):

Keep only captures with these content types, e.g. `text/html`, `application/pdf`, `image/*`. Empty = all.

## `urlRegex` (type: `string`):

Java regex that must match the **whole** original URL, e.g. `.*\.pdf` or `.*/blog/.*`. Applied by the Internet Archive server, so filtered URLs are never downloaded or charged.

## `excludeUrlRegex` (type: `string`):

Drop URLs whose whole original URL matches, e.g. `.*\.(css|js|png|jpg|gif|svg|woff2?)`.

## `skipQueryStrings` (type: `boolean`):

Drop URLs containing `?` (tracking parameters, search pages, session IDs). Useful for clean page lists.

## `skipTrackingUrls` (type: `boolean`):

Drop URL variants whose query contains analytics or session parameters (`utm_*`, `__hstc`, `fbclid`, `gclid`, `sessionid`, `PHPSESSID`…). Real dynamic pages such as `?id=12` or `?p=345` are kept. Filtered on the Internet Archive server, never charged.

## `skipMalformedUrls` (type: `boolean`):

Drop obviously broken URLs the archive crawler picked up from bad HTML (containing `"`, `<`, `>`, `%22`, braces or whitespace). Not charged.

## `captureStats` (type: `boolean`):

On: reads every capture to give `firstCapture`, `lastCapture`, `captureCount`, `uniqueVersions` (distinct content digests), the latest status and a direct link to the latest snapshot. Off: one CDX row per URL (collapse by urlkey), faster for huge domains, but only the first capture is known.

## `maxCapturesPerUrl` (type: `integer`):

Home pages can have 100,000+ captures. After this many, counting stops for that URL (`captureCountCapped: true`), the newest capture is looked up directly and the scan moves on. Keeps runs fast.

## `fetchContent` (type: `boolean`):

Download each URL's archived page (raw `id_` mode, without the Wayback toolbar) and add its title and text. Charged per snapshot downloaded (`snapshot-fetched`). Non-text captures (images, PDFs) are skipped and not charged.

## `contentFormat` (type: `string`):

`markdown`: main content as clean Markdown with links (good for LLMs). `text`: plain text. `html`: the raw archived HTML.

## `contentSnapshot` (type: `string`):

Content of the newest successful (HTTP 200) capture, or the earliest one.

## `maxContentChars` (type: `integer`):

Content longer than this is cut (`contentTruncated: true`).

## `compareFrom` (type: `string`):

Set both compare dates to get a content diff per URL: the capture closest to each date is downloaded, converted to Markdown and compared (similarity, lines added/removed, unified diff). Up to 2 `snapshot-fetched` events per URL. Use exact page URLs or a small `Max URLs`. Formats: `2020`, `2020-06-30`, `2 years`.

## `compareTo` (type: `string`):

The newer date for the diff, e.g. `2025-01-01` or `1 day`.

## `maxDiffChars` (type: `integer`):

The unified diff is cut after this many characters (`diffTruncated: true`). `addedLines` / `removedLines` keep up to 200 lines each.

## `cdxDelaySecs` (type: `integer`):

Minimum pause between Internet Archive CDX API calls. The Actor already slows down automatically on HTTP 429/503 and retries with backoff; raise this if you run many jobs in parallel.

## `requestTimeoutSecs` (type: `integer`):

Timeout for downloading one archived snapshot. Failed requests are retried with backoff.

## Actor input object example

```json
{
  "urls": [
    "crawlee.dev"
  ],
  "matchType": "auto",
  "maxUrls": 200,
  "maxUrlsPerInput": 0,
  "statusCodes": [
    "200"
  ],
  "mimeTypes": [],
  "skipQueryStrings": false,
  "skipTrackingUrls": true,
  "skipMalformedUrls": true,
  "captureStats": true,
  "maxCapturesPerUrl": 5000,
  "fetchContent": false,
  "contentFormat": "markdown",
  "contentSnapshot": "latest",
  "maxContentChars": 100000,
  "maxDiffChars": 20000,
  "cdxDelaySecs": 1,
  "requestTimeoutSecs": 60
}
```

# Actor output Schema

## `urls` (type: `string`):

No description

## `content` (type: `string`):

No description

## `diff` (type: `string`):

No description

## `errors` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "crawlee.dev"
    ],
    "maxUrls": 200,
    "statusCodes": [
        "200"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("gazidev/wayback-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["crawlee.dev"],
    "maxUrls": 200,
    "statusCodes": ["200"],
}

# Run the Actor and wait for it to finish
run = client.actor("gazidev/wayback-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "crawlee.dev"
  ],
  "maxUrls": 200,
  "statusCodes": [
    "200"
  ]
}' |
apify call gazidev/wayback-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gazidev/wayback-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/X5Q6fTWa0yRcXN2xF/builds/PVDhTmFbCci6qFe09/openapi.json