Wayback Machine CDX Scraper

---

🎯 What this scrapes

The Internet Archive publishes a CDX Server API (web.archive.org/cdx/search/cdx) that indexes every snapshot it has ever taken of a URL, host, or domain — timestamp, original URL, MIME type, HTTP status, content digest, and length. This Actor drives that API for you: it builds the right matchType/filter/collapse/date-range query, walks resumeKey pagination across tens of thousands of rows without losing a page, and writes one clean row per capture. Point it at a domain to rebuild your full historical URL inventory (handy after a migration, when pages that used to rank 301 into oblivion), or at a single page to trace every version the Archive ever saw.

🔥 What we handle for you

• 🔁 We retry with exponential backoff on 408 / 429 / 503 and honour Retry-After — up to 5 attempts per page.
• 🧭 We paginate correctly — resumeKey chained page by page, stopping the instant a short page or your maxResults cap says we're done, so you never get truncated or duplicated rows.
• 🧱 We fault-isolate bad pages — one page that fails after retries gets skipped and logged; it never kills the whole run.
• 🧊 We keep the dataset clean — Pydantic-validated rows, fixed-position column mapping (never guessed from headers), ISO-8601 timestamps.
• 💰 You pay only for results that land. No data → no charge beyond the small actor-start warm-up fee.

💡 Use cases

• Post-migration URL recovery — rebuild the full list of URLs a domain ever had so you can 301 every one of them correctly instead of guessing.
• OSINT / domain diligence — reconstruct a site's publishing history: when pages appeared, what they served, when they changed.
• Historical content audits — filter by status code and MIME type to find every HTML page that was once live under a domain.
• Change monitoring — pull digests over a date range to see exactly when a page's content last changed.

⚙️ How to use it

1. Click Try for free at the top of the page.
2. Fill in the input form — most fields have sensible defaults.
3. Click Start. Output streams into the run's dataset.
4. Export from Storage → Dataset as JSON, CSV, or Excel — or fetch via the API.

📥 Input

Field	Type	Required	Default	Notes
url	string	yes	'stripe.com'	Target URL, host, or domain to look up in the Wayback Machine CDX index, e.g. "stripe.com" or…
matchType	string	no	'domain'	How the URL is matched: an exact URL, a URL prefix, an exact host, or the whole domain (every subdomain and path — the…
dateFrom	string	no	'—'	Start of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no lower…
dateTo	string	no	'—'	End of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no upper…
filters	array	no	[]	Repeatable CDX filter expressions, e.g. "statuscode:200", "!statuscode:404", "mimetype:text/html".
collapse	string	no	'—'	CDX collapse expression to de-duplicate adjacent rows sharing a field, e.g. "urlkey" or "timestamp:8". Leave blank to…
pageSize	integer	no	1000	Rows requested per CDX page (the API's "limit" parameter).
maxResults	integer	no	10000	Hard cap on total rows fetched across all pages — protects spend and compute.
proxyConfiguration	object	no	{'useApifyProxy': False}	web.archive.org is keyless and not known to fingerprint clients at reasonable volumes — proxy is off by default. Switch…

Example input

{
  "url": "stripe.com",
  "matchType": "domain",
  "pageSize": 5,
  "maxResults": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

📤 Output

Every row is one dataset item.

Field	Type	Notes
urlkey	string	SURT canonical key for the captured URL (CDX column 0).
timestamp	string	Raw 14-digit capture timestamp (CDX column 1).
capture_date	string	capture timestamp converted to ISO 8601 UTC.
original_url	string	The URL as originally captured (CDX column 2).
mimetype	['string', 'null']	Content MIME type at capture time (CDX column 3), null when the Archive recorded "-".
statuscode	['string', 'null']	HTTP status code at capture time (CDX column 4), null when the Archive recorded "-".
digest	string	Content digest / checksum for this capture (CDX column 5).
length	['integer', 'null']	Captured response length in bytes (CDX column 6), null when not numeric.
archive_url	string	Direct playback URL for this snapshot on web.archive.org.
scraped_at	string	When this row was recorded by the Actor, ISO 8601 UTC.

Example output

{
  "urlkey": "com,stripe)/",
  "timestamp": "20260815120133",
  "capture_date": "2026-08-15T12:01:33+00:00",
  "original_url": "https://stripe.com/",
  "mimetype": "text/html",
  "statuscode": "200",
  "digest": "3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ",
  "length": 18452,
  "archive_url": "https://web.archive.org/web/20260815120133/https://stripe.com/",
  "scraped_at": "2026-09-01T12:00:00+00:00"
}

💰 Pricing

Pay-Per-Event — you pay only when these events fire:

Event	USD	What it is
actor-start	$0.05	One-off warm-up charge per run
snapshot-scraped	$0.002	PPE event

Example: 1 000 results at the rates above ≈ $2.05. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

🚧 Limitations

This Actor covers the CDX Server API's index only — it does not fetch archived page content, use the plain-text output=text mode, expose a custom field list, or touch the Wayback Machine's save/write endpoints. maxResults is our own spend/compute safety cap, not an Internet Archive limit.

❓ FAQ

Does this fetch the archived page content?

No — this Actor returns the CDX index (capture metadata: timestamp, status, MIME type, digest, length), not rendered HTML. Each row's archive_url is a direct link you can fetch yourself or feed into a follow-up scrape.

What does "domain" match type actually match?

domain returns every URL under the target domain across all subdomains and paths — the flagship "recover my whole site's history" use case. Use host to stay on one hostname, prefix for a URL prefix, or exact for one exact URL.

Why are some statuscode or length values null?

The Internet Archive records - for some captures (redirect-only or malformed responses). The Actor converts those to null rather than treating them as errors.

What happens if my query matches nothing?

The run finishes successfully with zero rows and a status message describing exactly what was searched — a narrow filter with no matches is not a failure.

💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an issue on the Actor's Issues tab on Apify Console — we ship fixes weekly and we read every report.

---