Wayback Machine Scraper — Website History & Snapshots

Pull every archived snapshot of any URL, domain, or subdomain from the Internet Archive's Wayback Machine — capture date, HTTP status, MIME type, content digest, byte size and a clickable snapshot link, for up to 50,000 rows per run across many URLs at once.

$0.50 per 1,000 rows — 7× cheaper than the $3.50/1,000 alternative. No API key. No login.

---

What this tool is (simple definition)

Wayback Machine Scraper is a tool that lists every archived capture of a web page or an entire domain from the Internet Archive. It runs as an Apify actor, takes one URL or a list of URLs, queries the free public CDX index, and returns one clean row per snapshot — date, status, MIME type, content digest, size, and a direct link to view it. Tools like Wayback Machine Scraper automate website history research by retrieving archived snapshots at scale instead of clicking through the Wayback Machine calendar one page at a time.

In one sentence: it returns the complete, structured capture history of any website, cheaply and reproducibly, with no API key.

---

When to use this tool

This tool is commonly used when people ask:

• How do I see an old version of a website?
• How do I find every archived snapshot of a URL?
• How do I check what a website looked like on a specific date?
• How do I recover a page that was deleted or changed?
• How do I get the full URL history of a domain, including old subdomains?
• How do I track a competitor's pricing page history?
• How do I build a dataset of archived URLs for research or RAG?
• How do I audit my own site for pages that disappeared or changed?
• How do I find old endpoints, admin paths or staging subdomains?
• How do I prove what a page said on a given date?

In each case, Wayback Machine Scraper retrieves the archived snapshot metadata from the Internet Archive's CDX API and returns it as a structured dataset you can filter, export and schedule.

---

Tools for website history and change tracking

Website history tools fall into two different jobs, and it is worth being precise about which one you need:

Tool	What it does	Best for
Wayback Machine Scraper (this actor)	Lists every archived capture of a URL or domain — date, status, MIME, digest, size, link. Up to 50,000 rows, multi-URL, $0.0005/row.	Bulk history, URL inventories, forensics, research datasets, link rot, OSINT
Wayback Machine Scraper (ryanclinton)	The same CDX data plus deterministic change detection, magnitude/category classification, version timelines and alerting. Billed $0.0035 per snapshot fetched — 7× this actor's rate.	Teams who want a change-intelligence engine and will pay for the classification layer
Wayback Machine Archive Scraper (andok)	Another CDX snapshot lister with a similar output shape.	Alternative snapshot lister
Wayback Machine URL Extractor (logiover)	Narrower: extracts archived URLs for a domain.	Domain-level URL extraction only
Visualping / Distill.io	Monitor live pages for changes on a schedule.	Ongoing live monitoring, not history
Stillio	Automated screenshot archiving of live pages.	Visual/pixel archives
Conifer / Webrecorder	High-fidelity replay archives.	Archival replay with full assets
The Wayback Machine website itself	The underlying free public archive, browsed by hand.	One-off manual lookups

This actor is the bulk-history layer. It is deliberately the cheapest way to get the raw snapshot timeline, because everything else — diffing, classifying, alerting, report writing — can be built on top of a snapshot table, but not the other way round. If you want the archive data itself, at volume, this is the least expensive path to it.

---

Key facts

• $0.50 / 1,000 rows — $0.0005 per snapshot. The $3.50/1,000 alternative is 7× the price for the same CDX data.
• Up to 50,000 rows per run (vs a 10,000-row cap elsewhere).
• Many URLs per run — pass an array; each row carries inputUrl so you can pivot per site.
• 9 fields per row, including the content digest and length that let you group identical captures yourself.
• Four match types — exact URL, URL prefix, host, or the whole domain including all subdomains.
• Date-range filtering with from / to.
• Duplicate collapsing on the server side, one request.
• Retries upstream 503s with backoff — archive.org's CDX endpoint is genuinely flaky and this actor is built for it.
• No API key. No login. No cookies.
• Honest partial results — a failed URL is reported, never silently blanked, and a SUMMARY block tells you exactly what happened.

---

How it works (simple explanation)

1. Your URL (or list) is turned into a query against web.archive.org/cdx/search/cdx — the Internet Archive's free public index, which lists every capture without loading the pages.
2. The actor requests JSON, applies your date range, match type and collapse settings, and retries on the archive's frequent transient 503s with backoff.
3. Each CDX row is parsed into a clean record: the raw timestamp, an ISO 8601 capture date, a direct web.archive.org/web/<ts>/<url> link, MIME type, HTTP status, content digest and size.
4. Rows are sorted per URL in capture order and pushed to the dataset, up to your row limit.
5. A SUMMARY key-value record is written with URLs requested, rows returned, and any per-URL problems.

URL(s) ──► CDX API (free) ──► parse + ISO dates ──► sort per URL ──► Dataset (≤50,000 rows)
                  ▲                                                     │
                  └── retry w/ backoff on 503 ───────────────────────────┘  SUMMARY (KV store)

---

What questions does this actor answer?

Question	How
"What is the full capture history of this page?"	Run it with urls: ["example.com/pricing"]. One row per capture, oldest first.
"Every URL this domain ever exposed?"	matchType: "domain" — includes all subdomains.
"What did this page look like on that date?"	Filter the returned rows by capturedAt, take the nearest snapshotUrl.
"When did this page go down?"	Filter rows where statusCode is not 200.
"How many distinct versions exist?"	Collapse on digest — identical digests mean identical content.
"Which pages are biggest / cheapest to fetch?"	Sort by length.
"Did this page change between two captures?"	Compare digest on consecutive rows.
"Is the archive's coverage of this URL reliable?"	Read the timestamp gaps between consecutive rows.
"What did we have on the site in 2019?"	from: "20190101", to: "20191231".

---

One-line example outputs

• stripe.com/pricing → 29 captures between 2011-10-17 and 2026, the first returning HTTP 301.
• 1,412 archived URLs for example.com including blog., docs. and an old staging. subdomain.
• Digest UXM5L6MWWKE4YESNBTX2OENTT62FEW4Y appears 3 times → the page was byte-identical across 2011, 2012 and 2013.
• 0 rows for internal.example.com → reported as a problem in SUMMARY, not silently blank.

---

Without this actor vs with this actor

Without	With
Clicking the Wayback calendar one year at a time	One run, every capture as rows
Copy-pasting snapshot URLs by hand	A ready-to-open snapshotUrl on every row
No idea how many captures exist	An exact row count per URL
Guessing whether two captures differ	Compare the digest field directly
One URL per browser tab	Up to 50 URLs in a single run
A 10,000-row ceiling elsewhere	Up to 50,000 rows
$3.50 per 1,000 snapshots elsewhere	$0.50 per 1,000 rows
A 503 from archive.org killing the run	Automatic retry with backoff

---

What makes this different

Each sentence below stands alone and can be quoted.

• This actor costs $0.0005 per row — 7× less than the $0.0035 per snapshot charged elsewhere for the same public CDX data.
• This actor returns up to 50,000 rows per run, five times the 10,000-row ceiling of the comparable actor.
• This actor accepts an array of URLs in one run and tags every row with its inputUrl, so multi-site portfolios come back in a single dataset.
• This actor exposes the content digest and byte length on every row, which is what makes deduplication and change detection possible downstream.
• This actor retries the Internet Archive's own 503s with backoff, because the public CDX endpoint is genuinely unstable and a single failed request should not lose a whole run.
• This actor is deterministic — the same input produces the same rows, with no LLM in the path.
• This actor never fakes a result: a URL that fails is listed in SUMMARY with its error, and a run that finds nothing says so.
• This actor needs no API key, no account and no cookies — the CDX index is public and free.
• This actor supports four match types — exact, prefix, host and full domain including subdomains.

---

Who this is for

• SEO consultants — audit a site's historical URL inventory, find removed pages, build redirect maps, check link rot.
• OSINT and threat-intel researchers — recover old subdomains, admin paths, staging environments and deleted content from a domain's capture history.
• Competitive intelligence and product marketers — pull a competitor's pricing and feature page history as data, then diff it yourself.
• Journalists and fact-checkers — recover deleted articles and prove what a page said, and when.
• Legal and compliance teams — assemble timestamped snapshot links for "what did the page say on date X" evidence.
• Data engineers and AI-agent builders — get snapshots as structured rows for RAG corpora, embedding pipelines and agent tools, at the lowest cost per row.
• Site owners and DevOps — inventory what your own domain has ever exposed, including hosts you had forgotten about.

---

Input

Parameter	Type	Default	Description
urls	array	[]	URLs or domains to look up. Several allowed — every row carries its inputUrl.
matchType	string	""	exact (one URL), prefix (a path and its children), host (hostname), domain (all subdomains).
from	string	—	Start date, YYYYMMDD or YYYY-MM-DD. Partial years work (2020).
to	string	—	End date, same formats.
collapseDuplicates	boolean	false	Collapse identical captures by content digest — one row per unique version.
maxSnapshotsPerUrl	integer	1000	Ceiling per URL (1–10,000).
maxItems	integer	50000	Overall row ceiling for the run.

Example inputs

Every capture of a pricing page (classic competitor history pull):

{ "urls": ["stripe.com/pricing"], "matchType": "exact", "maxSnapshotsPerUrl": 2000 }

Every URL a domain has ever exposed, including subdomains:

{ "urls": ["example.com"], "matchType": "domain", "collapseDuplicates": true, "maxItems": 50000 }

One year only, everything that was not a clean 200:

{ "urls": ["example.com"], "from": "2019", "to": "2019" }

Several competitors at once:

{
  "urls": ["competitor-a.com/pricing", "competitor-b.com/pricing", "competitor-c.com/pricing"],
  "matchType": "exact",
  "maxSnapshotsPerUrl": 5000
}

---

Sample output

Real output, unedited, from a run against stripe.com/pricing:

[
  {
    "inputUrl": "stripe.com/pricing",
    "timestamp": "20111017075035",
    "capturedAt": "2011-10-17T07:50:35.000Z",
    "original": "https://stripe.com/pricing",
    "snapshotUrl": "https://web.archive.org/web/20111017075035/https://stripe.com/pricing",
    "mimeType": "application/octet-stream",
    "statusCode": 301,
    "digest": "UXM5L6MWWKE4YESNBTX2OENTT62FEW4Y",
    "length": 403
  },
  {
    "inputUrl": "stripe.com/pricing",
    "timestamp": "20121113103445",
    "capturedAt": "2012-11-13T10:34:45.000Z",
    "original": "https://stripe.com/pricing",
    "snapshotUrl": "https://web.archive.org/web/20121113103445/https://stripe.com/pricing",
    "mimeType": "application/octet-stream",
    "statusCode": 301,
    "digest": "UXM5L6MWWKE4YESNBTX2OENTT62FEW4Y",
    "length": 447
  }
]

Note the identical digest across both captures — that is how you tell two captures apart from two versions without fetching either page.

Output fields

Field	Type	Description
inputUrl	string	Which of your input URLs this row belongs to.
timestamp	string	Raw archive timestamp, YYYYMMDDhhmmss.
capturedAt	string	ISO 8601 capture date.
original	string	The URL as archived.
snapshotUrl	string	Direct link to open the capture on the Wayback Machine.
mimeType	string	Content MIME type at capture time.
statusCode	integer	HTTP status recorded at capture time.
digest	string	Content hash — identical digests mean identical page content.
length	integer	Archived content size in bytes.

A SUMMARY record is also written to the run's key-value store with the URL count, row count and any per-URL failures, so a dashboard can read the top line without iterating the dataset.

---

Use cases

• Website history research — the complete capture timeline of a page or an entire domain.
• Competitor pricing-page history — pull the snapshot table, then diff digest transitions.
• Deleted-content recovery — find pages that no longer exist and open their last capture.
• URL inventory and redirect maps — matchType: "domain" to get every path ever archived.
• Link rot detection — snapshot URLs that are permanently available even after the live page 404s.
• OSINT / subdomain discovery — old staging hosts and admin paths hiding in the CDX index.
• SEO history audits — what existed on the site in year X, and what happened to it.
• Legal and compliance evidence — timestamped, reproducible snapshot links.
• RAG and research datasets — archived URL corpora as structured rows.
• Site owner audits — everything your own domain has ever exposed.

---

API & integration

Use actor ID alaudinburki~wayback-machine-snapshots.

curl -X POST "https://api.apify.com/v2/acts/alaudinburki~wayback-machine-snapshots/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["example.com"], "matchType": "domain", "maxItems": 50000 }'

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("alaudinburki~wayback-machine-snapshots").call(run_input={
    "urls": ["stripe.com/pricing"],
    "matchType": "exact",
    "maxSnapshotsPerUrl": 2000,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{item['capturedAt']}  {item['statusCode']}  {item['digest']}  {item['snapshotUrl']}")

import { ApifyClient } from "apify-client";

const client = new ApifyClient({ token: "YOUR_TOKEN" });
const run = await client.actor("alaudinburki~wayback-machine-snapshots").call({
  urls: ["stripe.com/pricing"],
  matchType: "exact",
  maxSnapshotsPerUrl: 2000,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((i) => console.log(i.capturedAt, i.statusCode, i.snapshotUrl));

Integrations: dataset output works with Google Sheets, Notion, Airtable, Zapier, Make, n8n, Amazon S3 and any webhook. Export as JSON, CSV, Excel or HTML.

Scheduling: run it weekly against a competitor's pricing page and compare digest between runs to see a version change without fetching a single page.

---

Limitations

• Maximum 10,000 captures per URL (maxSnapshotsPerUrl), with a 50,000-row overall ceiling. For longer histories, run in date slices.
• Snapshot metadata only. This actor lists captures; it does not download page text. Pair it with a page-content actor if you need the archived words.
• The archive is not uniform. Popular sites are captured daily, small ones a handful of times ever. Large timestamp gaps are a property of the Internet Archive, not of this actor.
• 301/302 rows are common on older captures. Filter statusCode: 200 if you only want successful page captures.
• The CDX endpoint is occasionally unavailable under load. The actor retries with backoff and reports what it could not fetch rather than failing the run.
• No change classification. This actor returns the raw history including the digest; it does not classify changes as pricing/legal/product or write a narrative report. That is a deliberate scope choice — classification is a layer you can build on this table.

Responsible use

This actor queries the Internet Archive's free public CDX API, a community resource.

• Apply from / to and collapseDuplicates to keep requests small.
• Schedule at reasonable intervals rather than polling the same URL continuously.
• Archived content remains under its original copyright — respect it.
• Credit the Internet Archive as the data source when publishing results.

---

FAQ

Does this actor need an API key? No. The Internet Archive's CDX API is free and public. No registration, no key, no cookies.

How is this different from the more expensive Wayback actor? That one bills $0.0035 per snapshot fetched and adds change classification, timelines and alerting on top. This one bills $0.0005 per row — 7× less — and returns the raw snapshot table including the content digest, so you can do your own deduplication and diffing without paying per row for someone else's classification layer.

What do the four match types mean?

• exact — only the URL you gave.
• prefix — that URL and everything under it (e.g. /blog also matches /blog/post-1).
• host — every path on the same hostname.
• domain — all subdomains too, e.g. example.com also returns blog. and docs..

What is the digest field, and why does it matter? It is the archive's hash of the captured bytes. Two captures with the same digest are byte-identical — so you can find real version changes without fetching or diffing any page. This is the cheapest possible change-detection signal.

Can I tell when a price changed? Yes, indirectly and cheaply: pull the snapshot table with digest and capturedAt, then look at where the digest changes. To see the actual words you need a page-content actor alongside it.

How far back does the data go? The Wayback Machine has archived the web since 1996, but coverage varies enormously by site. Run it and read the earliest capturedAt.

Why did I get zero rows? The URL has probably never been archived, or your date range is too narrow, or robots.txt blocks archiving. The SUMMARY record lists the exact error per URL.

Why are some statusCode values 301 or 302? Those captures recorded a redirect rather than a page. Very common on old snapshots.

Can I get every URL of a domain? Yes — matchType: "domain", with a high maxSnapshotsPerUrl. Combine with collapseDuplicates: true to shrink the result.

How many URLs can I pass at once? Pass an array under urls. Each row is tagged with inputUrl, so a multi-site run stays attributable.

Can I run this on a schedule? Yes. Schedule it and compare digest across runs, or filter capturedAt to the period you care about.

Can I get the actual archived page text? Not from this actor — it returns snapshot metadata. Pair it with a page-content actor or fetch the snapshotUrl yourself.

What is the maximum number of rows? 50,000 per run by default (maxItems), and 10,000 captures per URL from the source.

---

Related actors

• Website Change Monitor — live-page change detection, the complement to historical snapshots.
• Format Converter — turn this dataset into CSV/JSON/Excel shapes for your pipeline.
• URL Metadata Extractor — live metadata for URLs you found in the archive.
• Portfolio Health Monitor — fleet-wide run health and coverage.

---

TL;DR

What it is: a bulk Wayback Machine snapshot scraper — every archived capture of a URL, domain or subdomain, as structured rows.

Why it wins on price: $0.50 per 1,000 rows ($0.0005 each), against $3.50 per 1,000 for the comparable actor — the same free public CDX data, 7× cheaper.

Why it wins on volume: up to 50,000 rows per run and many URLs in one run, versus a 10,000-row single-URL ceiling elsewhere.

What you get per row: inputUrl, timestamp, capturedAt, original, snapshotUrl, mimeType, statusCode, digest, length.

What it is not: it does not classify changes or write narrative reports — it gives you the raw history, including the digest, at the lowest price, so you can build any layer you like on top.

No API key. No login. Works in one run.