# Wayback Page Extractor (`espmsftw/wayback-page-extractor`) Actor

Extract clean text, metadata, offline HTML and images from Wayback Machine (archive.org) snapshots for a list of URLs and a date range. JSON dataset + one ZIP per snapshot.

- **URL**: https://apify.com/espmsftw/wayback-page-extractor.md
- **Developed by:** [ESPMS](https://apify.com/espmsftw) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 snapshot extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Wayback Page Extractor

Extract clean, machine-readable data from the **Wayback Machine** (archive.org) for a list of URLs. For each URL it queries the **CDX API**, picks the snapshot(s) in your date range, downloads the archived `id_` version, and returns **clean text + metadata + offline HTML + images**, one ZIP per snapshot.

> **Built by a supervised AI agent (ESPMS) under human supervision.** The code is reviewed and the publish decision is made by a human.

### What it does

1. **CDX lookup** — for each URL it calls the archive.org CDX API (`matchType`, `from`/`to` dates, `collapse`, `limit`) to find snapshots.
2. **Download** — fetches the archived `id_` (original, unmodified) version of each snapshot.
3. **Extract** — pulls clean text, page metadata (`title`, `description`, Open Graph, `canonical`, charset), and up to `maxImagesPerPage` images.
4. **Offline HTML** — rewrites the HTML so it renders without network, with local image paths.
5. **Package** — writes a ZIP per snapshot and pushes one record to the Dataset.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `urls` | text (one URL per line) | — | **Required.** The URLs to look up. |
| `fromDate` | date `YYYY-MM-DD` | — | Start of the date range (CDX `from`), e.g. `2020-01-01`. `YYYYMMDD` also works. |
| `toDate` | date `YYYY-MM-DD` | — | End of the date range (CDX `to`), e.g. `2024-12-31`. `YYYYMMDD` also works. |
| `matchType` | `exact` | `prefix` | `domain` | `exact` | CDX `matchType`. |
| `maxSnapshotsPerUrl` | integer (1–100) | `1` | How many captures per URL: the most recent ones in the range. |
| `collapse` | string | `timestamp:6` with a date range, `urlkey` without | CDX `collapse`. |
| `outputsText` | boolean | `true` | Clean text of the page. |
| `outputsMetadata` | boolean | `true` | Title, description, Open Graph, canonical, charset. |
| `outputsHtml` | boolean | `false` | Offline HTML (renders without network). |
| `outputsImages` | boolean | `false` | Download the page images into the ZIP. |
| `maxImagesPerPage` | integer (1–100) | `20` | Max images per snapshot (when `outputsImages` is on). |
| `timeoutMs` | integer (1000–60000) | `30000` | Per-request timeout to archive.org. |

**Example input:**

```json
{
  "urls": "https://www.example.com/about\nhttps://www.example.com/pricing",
  "fromDate": "2019-01-01",
  "toDate": "2024-12-31",
  "maxSnapshotsPerUrl": 1,
  "outputsText": true,
  "outputsMetadata": true,
  "outputsHtml": true,
  "outputsImages": true
}
```

### Output

Two places, both per snapshot:

**1. Dataset (JSONL)** — one record per snapshot:

```json
{
  "url": "https://www.example.com/about",
  "timestamp": "20231105120000",
  "statuscode": "200",
  "mimetype": "text/html",
  "digest": "abc123",
  "length": "18234",
  "text": "Clean text of the page…",
  "metadata": { "title": "About us", "description": "…", "og": {}, "canonical": "https://www.example.com/about" },
  "htmlPath": "index.html",
  "images": ["images/img_1_20231105120000.png"],
  "zipPath": "archive_20231105120000.zip",
  "kvZipKey": "example.com_about_20231105120000.zip",
  "kvHtmlKey": "example.com_about_20231105120000.html"
}
```

If a URL has no snapshots in CDX, the record is `{ "url": "…", "notArchived": true }` — that is a **result, not an error**, and it is **not charged**. Fields such as `text`, `metadata`, `htmlPath` and `images` appear only when the matching output is switched on.

If archive.org does not answer for a URL after the retries, that URL gets a record with an `error` field, it is **not charged**, and the run **carries on** with the next URL. With `matchType` `prefix` or `domain`, the captures can belong to different pages under that prefix.

**2. Key-Value store** — **one ZIP and one HTML per snapshot**, keyed by `safeName(url)_<timestamp>`:

- `safeName(url)_<timestamp>.zip` — the snapshot package. Inside: `text.txt` (if `outputs.text`), the offline HTML (`index.html`), and `images/img_N_<timestamp>.<ext>` (downloaded images).
- `safeName(url)_<timestamp>.html` — the offline HTML on its own.

The Dataset record points to both via `kvZipKey` and `kvHtmlKey`.

### Pricing

**$0.002 per extracted snapshot** — pay-per-event. URLs that are not archived are returned as `notArchived` and are **not charged**. Platform usage is included in the price.

### Honest limits

- **Rate limit:** about 1 request per second to archive.org, with backoff on 429/503 (2 → 4 → 8 s, up to 3 retries). Large batches are slow by design, to stay a good citizen of archive.org.
- **Content rights are yours to respect.** archive.org is a public repository; the content belongs to its authors. Use it for the purposes its terms allow.
- **`id_` snapshots only** — we download the original archived bytes, not the Wayback Machine's rewritten view.
- **No login, no CAPTCHA** — it only reads the public CDX API and public snapshots.

### Development

- Entry point for the platform: `src/entry.ts` (Actor.init / getInput / setValue / exit).
- Core pipeline: `src/main.ts` (portable, runs in tests and on-platform).
- Platform I/O (Dataset + KV store): `src/platform.ts`.
- Tests: `npx tsx --test test/`.

# Actor input Schema

## `urls` (type: `string`):

One URL per line.

## `fromDate` (type: `string`):

Earliest capture date, YYYY-MM-DD (YYYYMMDD also works). Empty = no lower limit.

## `toDate` (type: `string`):

Latest capture date, YYYY-MM-DD (YYYYMMDD also works). Empty = no upper limit.

## `matchType` (type: `string`):

exact (the URL as given), prefix (every URL under it) or domain (the whole domain).

## `collapse` (type: `string`):

CDX collapse parameter, e.g. timestamp:6 (one capture per month) or digest. Empty = timestamp:6 with a date range, urlkey without.

## `maxSnapshotsPerUrl` (type: `integer`):

How many captures to extract per URL (1–100): the most recent ones in the range.

## `outputsText` (type: `boolean`):

Include the clean text (text.txt) in the ZIP and the dataset.

## `outputsMetadata` (type: `boolean`):

Include title, description, Open Graph, canonical and charset.

## `outputsHtml` (type: `boolean`):

Include the offline HTML (links and images rewritten) in the ZIP and the key-value store.

## `outputsImages` (type: `boolean`):

Download the images referenced in the page into the ZIP.

## `maxImagesPerPage` (type: `integer`):

Image limit per capture (default 20, max 100).

## `timeoutMs` (type: `integer`):

Per-request timeout to archive.org, in milliseconds. Timeouts are retried 3 times (2, 4 and 8 s).

## Actor input object example

```json
{
  "matchType": "exact",
  "maxSnapshotsPerUrl": 1,
  "outputsText": true,
  "outputsMetadata": true,
  "outputsHtml": false,
  "outputsImages": false,
  "maxImagesPerPage": 20,
  "timeoutMs": 30000
}
```

# Actor output Schema

## `results` (type: `string`):

One item per URL and snapshot: clean text, metadata and the key of its ZIP, or notArchived / error for URLs without a usable snapshot.

## `files` (type: `string`):

One ZIP per extracted snapshot (offline HTML and, if requested, images), stored in the run's default key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("espmsftw/wayback-page-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("espmsftw/wayback-page-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call espmsftw/wayback-page-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,espmsftw/wayback-page-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/guXTGxvAXvqrxRxu3/builds/GMBfsSnVbbEPahq2Q/openapi.json