# Wayback Machine Scraper: Archived URLs, Snapshots & Page Text (`manageable_gully/wayback-machine-scraper`) Actor

Scrape the Internet Archive Wayback Machine via its official CDX API: every archived URL of a domain, snapshot history of any page (only content changes), first/last capture summary, and archived page text or Markdown. Date/status filters, alerts. $0.50 per 1,000 records.

- **URL**: https://apify.com/manageable_gully/wayback-machine-scraper.md
- **Developed by:** [WebIntel Tools](https://apify.com/manageable_gully) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 archive records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Machine Scraper: Archived URLs, Snapshots, History & Page Text

Get **Internet Archive Wayback Machine data as clean JSON, CSV or Excel** through the **official Wayback CDX Server API**: **every archived URL of a website** (a Wayback URL extractor), the **full snapshot history of any page** (optionally only the captures where the content changed), a **one-row archive summary** per URL (first and last capture, years archived, snapshot closest to a date), and the **archived page text or Markdown** of any capture.

No browser, no login, no proxy. **$0.50 per 1,000 records**, archived page content $2 per 1,000 pages.

### What you can use it for

- **SEO and site migrations:** list every URL a domain ever had to find lost pages, old blog posts and backlinks targets, and build 301 redirect maps.
- **Recover deleted content:** get the text or Markdown of pages that no longer exist on the live site.
- **Competitor research:** see how a competitor's pricing, homepage or product pages changed over time (only content changes, one per month, etc.).
- **Domain due diligence:** before buying an expired domain, check when it was first archived, how long it has been online and what it used to host.
- **Compliance and legal evidence:** timestamped Wayback links of what a page showed on a given date.
- **Security and OSINT on your own assets:** forgotten endpoints, old files and PDFs under a domain.
- **AI and RAG:** historical page text as clean Markdown for LLM pipelines or dataset building.

### How to use it

1. Choose a **mode**: `urls` (all archived URLs of a site), `snapshots` (capture history of pages) or `summary` (one row per URL).
2. Add **targets**: `example.com`, `https://example.com/pricing`, `example.com/blog/*` or a `web.archive.org/web/...` link.
3. Optional filters: date range, status codes (e.g. `200`, `3xx`), MIME types (e.g. `text/html`, `application/pdf`), URL regex.
4. Optional: turn on **Fetch archived page content** to get the title, meta description, H1, word count and text/Markdown.
5. Click **Start**, or schedule it with **Only new** to get alerts about newly archived pages or captures.

### Why this Wayback Machine scraper

- **Official API only:** the Internet Archive's public Wayback CDX Server and Availability APIs. Nothing breaks when the Wayback website changes.
- **Three tools in one:** URL extractor, snapshot history and archive summary, each with date, status, MIME and regex filters.
- **Only content changes:** `collapse: digest` returns a capture only when the page content changed. Or limit to one capture per hour, day, month or year.
- **Clean page content:** archived HTML is fetched in its original form (no Wayback toolbar), scripts and styles removed, as text, Markdown or HTML.
- **No 10,000 cap:** large domains are paged with the CDX resume key, so you can export hundreds of thousands of URLs.
- **Polite and reliable:** one request at a time, automatic back-off when the archive asks to slow down.
- **Cheap and fair:** $0.50 per 1,000 records. Targets that were never archived, already-seen items and failed content fetches are free. 256 MB memory.

### Modes

| Mode | One row per | Typical use |
|---|---|---|
| `urls` (default) | unique archived URL, with its first capture | URL extractor, lost pages, redirect maps |
| `snapshots` | capture of a page (exact URL or prefix) | change history, price/page tracking, evidence |
| `summary` | target | first/last capture, years archived, closest snapshot to a date |

### Input examples

Every archived HTML page of a domain (incl. subdomains) that returned 200:

```json
{ "mode": "urls", "targets": ["example.com"], "statusCodes": ["200"], "mimeTypes": ["text/html"], "maxItemsPerTarget": 10000 }
```

All archived PDFs under a path:

```json
{ "mode": "urls", "targets": ["example.com/docs/*"], "urlFilter": ".*\\.pdf$", "excludeAssets": false }
```

A competitor's pricing page, only when it changed, with the page text:

```json
{ "mode": "snapshots", "targets": ["https://www.example.com/pricing"], "collapse": "digest", "statusCodes": ["200"], "fetchContent": true, "contentFormat": "markdown" }
```

The latest 10 captures of a homepage:

```json
{ "mode": "snapshots", "targets": ["https://example.com/"], "order": "newest", "maxItemsPerTarget": 10 }
```

Archive summary for a list of domains, plus the snapshot closest to 1 Jan 2020:

```json
{ "mode": "summary", "targets": ["example.com", "apify.com", "python.org"], "closestDate": "2020-01-01" }
```

Weekly alert of newly archived URLs on a site (schedule it):

```json
{ "mode": "urls", "targets": ["example.com"], "onlyNew": true, "stateKey": "example-weekly" }
```

### Output

`urls` / `snapshots` row:

```json
{
  "type": "snapshot",
  "target": "https://www.python.org/",
  "url": "https://www.python.org/",
  "timestamp": "20200101011950",
  "capturedAt": "2020-01-01T01:19:50Z",
  "statusCode": 200,
  "mimeType": "text/html",
  "lengthBytes": 9999,
  "digest": "XMMYQPPTNAC3U4V5J6HCSU6TOMUUYBQK",
  "archiveUrl": "https://web.archive.org/web/20200101011950/https://www.python.org/",
  "rawArchiveUrl": "https://web.archive.org/web/20200101011950id_/https://www.python.org/",
  "urlKey": "org,python)/",
  "title": "Welcome to Python.org",
  "metaDescription": "The official home of the Python Programming Language",
  "h1": null,
  "wordCount": 1021,
  "content": "Python is a programming language that lets you work quickly and integrate systems more effectively...",
  "scrapedAt": "2026-10-23T08:00:00Z"
}
```

`summary` row:

```json
{
  "type": "summary",
  "target": "python.org",
  "archived": true,
  "firstCapturedAt": "1997-05-01T01:16:26Z",
  "lastCapturedAt": "2026-10-23T05:46:27Z",
  "lastStatusCode": 200,
  "yearsCaptured": [1997, 1998, 1999, "..."],
  "yearsCount": 30,
  "archiveSpanDays": 10767,
  "closestCapturedAt": "2020-01-01T23:40:54Z",
  "closestArchiveUrl": "https://web.archive.org/web/20200101234054/https://www.python.org/",
  "calendarUrl": "https://web.archive.org/web/*/python.org"
}
```

Content fields (`title`, `metaDescription`, `canonicalUrl`, `h1`, `language`, `linksCount`, `wordCount`, `content`, `contentTruncated`) appear only with **Fetch archived page content**. A run summary (records, content pages, not-archived targets, API requests) is saved as `OUTPUT` in the key-value store.

### Pricing

Pay per event, no subscription:
| Event | Price |
|---|---|
| Archive record (URL, snapshot or summary) | $0.0005 ($0.50 per 1,000) |
| Archived page content (text/Markdown) | $0.002 ($2 per 1,000) |
| Actor start | $0.0003 |

Examples: 10,000 archived URLs of a domain is about $5. A year of monthly pricing-page snapshots with text is 12 x $0.0025 = $0.03. Set **Max cost per run** in the run options to cap spend; the Actor stops cleanly at the limit.

### FAQ

**Is it legal to use the Wayback Machine API?** The CDX Server and Availability APIs are public interfaces the Internet Archive provides for programmatic access. This Actor uses them politely (one request at a time, backs off on HTTP 429). Archived pages are third-party content: respect copyright and the original sites' rights when you reuse text.

**Does it collect personal data?** No. It returns URL and capture metadata and, on request, the text of archived pages you choose. Use the content option responsibly and comply with privacy law (e.g. GDPR) if pages contain personal information.

**Why are some URLs missing?** Only what the Internet Archive captured is available, and site owners can ask for exclusion. Some pages are archived only as redirects (3xx); filter with `statusCodes: ["200"]` to get real pages.

**How many URLs can I get for a big domain?** The CDX API pages through results with a resume key, so there is no fixed cap; use *Max records per target* to control cost.

**What does "Only content changed" do?** It collapses consecutive captures with the same content hash (digest), so you only get captures where the page actually changed.

**The archive is slow or returned 429.** The Wayback Machine is a free public service with rate limits. The Actor waits and retries automatically; if it still fails, the affected target gets an error row (not charged) and you can rerun later.

### Related tools by the same developer

- [RSS Feed Scraper & Reader](https://apify.com/manageable_gully/rss-feed-reader)
- [Web Page to Markdown for LLMs](https://apify.com/manageable_gully/page-to-markdown)
- [Website SEO Audit: SSL, DNS, Security Headers](https://apify.com/manageable_gully/website-health-audit)
- [Broken Link Checker](https://apify.com/manageable_gully/broken-link-checker)

# Actor input Schema

## `mode` (type: `string`):

urls = every unique archived URL of a domain or URL prefix with its first capture (find old/deleted pages, build redirect maps, SEO recovery). snapshots = every capture of the given pages over time (optionally only when the content changed). summary = one row per target: first and last capture, years archived, snapshot closest to a date.

## `targets` (type: `array`):

One per line: a domain (example.com), a URL (https://example.com/pricing), a URL prefix ending in \* (example.com/blog/\*), or a web.archive.org link.

## `matchType` (type: `string`):

Auto: in 'urls' mode a bare domain means the whole domain incl. subdomains and a URL with a path means that prefix; in 'snapshots'/'summary' mode it means the exact URL. A trailing \* always means prefix.

## `dateFrom` (type: `string`):

Only captures on/after this date: 2015, 2015-06, 2015-06-30, 20150630120000, or relative like '30 days' / '2 years'.

## `dateTo` (type: `string`):

Only captures on/before this date (same formats).

## `statusCodes` (type: `array`):

Keep only captures with these status codes, e.g. 200, 301, 404, or classes like 2xx / 3xx. Empty = all.

## `mimeTypes` (type: `array`):

Keep only these content types, e.g. text/html, application/pdf. Empty = all.

## `excludeAssets` (type: `boolean`):

In 'urls' mode, skip URLs of static assets (images, CSS, JS, fonts, media, archives).

## `urlFilter` (type: `string`):

Optional regular expression the full archived URL must match, e.g. .*/blog/.* or .\*.pdf$

## `collapse` (type: `string`):

Used in 'snapshots' mode. 'Only when content changed' skips captures identical to the previous one (same content hash).

## `order` (type: `string`):

Used in 'snapshots' mode.

## `closestDate` (type: `string`):

Used in 'summary' mode: also return the capture closest to this date (e.g. 2019-01-01).

## `fetchContent` (type: `boolean`):

Download each archived HTML page (original, without the Wayback toolbar) and add title, meta description, canonical, H1, word count and the text/Markdown. Charged per page.

## `contentFormat` (type: `string`):

Format of the 'content' field.

## `maxContentChars` (type: `integer`):

Truncate content to this many characters (0 = no limit).

## `maxContentPages` (type: `integer`):

Stop fetching content after this many pages (records are still saved).

## `maxItemsPerTarget` (type: `integer`):

0 = no limit (large domains can have millions of archived URLs).

## `maxItems` (type: `integer`):

0 = no limit.

## `onlyNew` (type: `boolean`):

Remember what earlier runs returned and save only new archived URLs ('urls' mode) or new captures ('snapshots' mode). Use with a schedule to get alerts.

## `stateKey` (type: `string`):

Separate memory for different scheduled tasks using Only new.

## Actor input object example

```json
{
  "mode": "urls",
  "targets": [
    "books.toscrape.com"
  ],
  "matchType": "auto",
  "excludeAssets": true,
  "collapse": "none",
  "order": "oldest",
  "fetchContent": false,
  "contentFormat": "text",
  "maxContentChars": 20000,
  "maxContentPages": 100,
  "maxItemsPerTarget": 100,
  "maxItems": 0,
  "onlyNew": false,
  "stateKey": "default"
}
```

# Actor output Schema

## `results` (type: `string`):

One row per archived URL, snapshot or summary: URL, capture time, status, MIME type, size, Wayback links, and optional page title/text.

## `summary` (type: `string`):

Records and content pages saved, targets not archived, already-seen items skipped and API requests made.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targets": [
        "books.toscrape.com"
    ],
    "maxItemsPerTarget": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("manageable_gully/wayback-machine-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "targets": ["books.toscrape.com"],
    "maxItemsPerTarget": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("manageable_gully/wayback-machine-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targets": [
    "books.toscrape.com"
  ],
  "maxItemsPerTarget": 100
}' |
apify call manageable_gully/wayback-machine-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,manageable_gully/wayback-machine-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/NiQCSGUYYdPdjC0KC/builds/sMfZpvCuHNhWPkx6z/openapi.json