Wayback Page Extractor

Extract clean, machine-readable data from the Wayback Machine (archive.org) for a list of URLs. For each URL it queries the CDX API, picks the snapshot(s) in your date range, downloads the archived id_ version, and returns clean text + metadata + offline HTML + images, one ZIP per snapshot.

Built by a supervised AI agent (ESPMS) under human supervision. The code is reviewed and the publish decision is made by a human.

What it does

1. CDX lookup — for each URL it calls the archive.org CDX API (matchType, from/to dates, collapse, limit) to find snapshots.
2. Download — fetches the archived id_ (original, unmodified) version of each snapshot.
3. Extract — pulls clean text, page metadata (title, description, Open Graph, canonical, charset), and up to maxImagesPerPage images.
4. Offline HTML — rewrites the HTML so it renders without network, with local image paths.
5. Package — writes a ZIP per snapshot and pushes one record to the Dataset.

Input

Field	Type	Default	Notes
urls	text (one URL per line)	—	Required. The URLs to look up.
fromDate	date YYYY-MM-DD	—	Start of the date range (CDX from), e.g. 2020-01-01. YYYYMMDD also works.
toDate	date YYYY-MM-DD	—	End of the date range (CDX to), e.g. 2024-12-31. YYYYMMDD also works.
matchType	exact | prefix | domain	exact	CDX matchType.
maxSnapshotsPerUrl	integer (1–100)	1	How many captures per URL: the most recent ones in the range.
collapse	string	timestamp:6 with a date range, urlkey without	CDX collapse.
outputsText	boolean	true	Clean text of the page.
outputsMetadata	boolean	true	Title, description, Open Graph, canonical, charset.
outputsHtml	boolean	false	Offline HTML (renders without network).
outputsImages	boolean	false	Download the page images into the ZIP.
maxImagesPerPage	integer (1–100)	20	Max images per snapshot (when outputsImages is on).
timeoutMs	integer (1000–60000)	30000	Per-request timeout to archive.org.

Example input:

{
  "urls": "https://www.example.com/about\nhttps://www.example.com/pricing",
  "fromDate": "2019-01-01",
  "toDate": "2024-12-31",
  "maxSnapshotsPerUrl": 1,
  "outputsText": true,
  "outputsMetadata": true,
  "outputsHtml": true,
  "outputsImages": true
}

Output

Two places, both per snapshot:

1. Dataset (JSONL) — one record per snapshot:

{
  "url": "https://www.example.com/about",
  "timestamp": "20231105120000",
  "statuscode": "200",
  "mimetype": "text/html",
  "digest": "abc123",
  "length": "18234",
  "text": "Clean text of the page…",
  "metadata": { "title": "About us", "description": "…", "og": {}, "canonical": "https://www.example.com/about" },
  "htmlPath": "index.html",
  "images": ["images/img_1_20231105120000.png"],
  "zipPath": "archive_20231105120000.zip",
  "kvZipKey": "example.com_about_20231105120000.zip",
  "kvHtmlKey": "example.com_about_20231105120000.html"
}

If a URL has no snapshots in CDX, the record is { "url": "…", "notArchived": true } — that is a result, not an error, and it is not charged. Fields such as text, metadata, htmlPath and images appear only when the matching output is switched on.

If archive.org does not answer for a URL after the retries, that URL gets a record with an error field, it is not charged, and the run carries on with the next URL. With matchType prefix or domain, the captures can belong to different pages under that prefix.

2. Key-Value store — one ZIP and one HTML per snapshot, keyed by safeName(url)_<timestamp>:

• safeName(url)_<timestamp>.zip — the snapshot package. Inside: text.txt (if outputs.text), the offline HTML (index.html), and images/img_N_<timestamp>.<ext> (downloaded images).
• safeName(url)_<timestamp>.html — the offline HTML on its own.

The Dataset record points to both via kvZipKey and kvHtmlKey.

Pricing

$0.002 per extracted snapshot — pay-per-event. URLs that are not archived are returned as notArchived and are not charged. Platform usage is included in the price.

Honest limits

• Rate limit: about 1 request per second to archive.org, with backoff on 429/503 (2 → 4 → 8 s, up to 3 retries). Large batches are slow by design, to stay a good citizen of archive.org.
• Content rights are yours to respect. archive.org is a public repository; the content belongs to its authors. Use it for the purposes its terms allow.
• id_ snapshots only — we download the original archived bytes, not the Wayback Machine's rewritten view.
• No login, no CAPTCHA — it only reads the public CDX API and public snapshots.

Development

• Entry point for the platform: src/entry.ts (Actor.init / getInput / setValue / exit).
• Core pipeline: src/main.ts (portable, runs in tests and on-platform).
• Platform I/O (Dataset + KV store): src/platform.ts.
• Tests: npx tsx --test test/.