Wayback Machine Scraper: Archived Snapshots & Pages

List every Wayback Machine snapshot of a URL or a whole domain from the Internet Archive, with capture date, original HTTP status, MIME type and content digest, and optionally download the archived page itself as clean text or raw HTML. Filter by date range, status code, MIME type and URL regex, de-duplicate to one capture per day/month/year or only captures where the content changed, or jump straight to the snapshot closest to a date.

Uses the Internet Archive's official public APIs (CDX server, availability API, raw id_ captures). No browser, no login, no start fee.

What you can do with it

• Find deleted pages: match a whole domain, collapse by URL and get every page ever archived, including ones that no longer exist (404s, old blog posts, removed products).
• Expired-domain and SEO research: see how long a domain has been live, what it hosted, when it changed hands, and how many pages it had.
• Content and price history: collapse by content changed to get only captures where a page actually changed, then fetch them to compare copy, pricing or terms over time.
• Competitor monitoring: what did a competitor's pricing or landing page say on a given date? Use Closest snapshot to date.
• Recover lost content: download archived text or HTML of pages from a dead site.

Input

Field	What it does
urls	Page URLs or domains, one per line.
matchType	exact (default), prefix, host or domain (host + subdomains).
from / to	Date range: YYYY, YYYY-MM or YYYY-MM-DD.
collapse	digest (content changed), hour, day, month, year, or url (one per unique URL).
statusCodes, mimeType, urlRegex	Server-side filters, e.g. ["200"], text/html, .*/product/.*.
newestFirst	Latest captures first.
closestTo	Return only the single capture closest to this date, per URL.
fetchContent, maxPagesFetched, contentFormat, maxContentChars	Download archived pages (status-200 captures) as text or HTML.
maxSnapshotsPerUrl, maxItems	Caps (and cost control). maxItems is shared fairly: every URL gets an equal share, and URLs with fewer captures pass the rest on.

Examples:

{ "urls": ["example.com"], "collapse": "year", "from": "2005", "to": "2015" }

{ "urls": ["oldsite.com"], "matchType": "domain", "collapse": "url", "statusCodes": ["200"], "mimeType": "text/html" }

{ "urls": ["https://www.competitor.com/pricing"], "collapse": "digest", "fetchContent": true, "maxPagesFetched": 20 }

Output

One item per snapshot:

{
  "inputUrl": "https://www.python.org/", "url": "http://www.python.org:80/",
  "timestamp": "20120603061049", "archivedAt": "2012-06-03T06:10:49Z",
  "snapshotUrl": "https://web.archive.org/web/20120603061049/http://www.python.org:80/",
  "rawUrl": "https://web.archive.org/web/20120603061049id_/http://www.python.org:80/",
  "statusCode": 200, "mimeType": "text/html", "digest": "…", "length": 5810, "urlkey": "org,python)/",
  "title": "Python Programming Language – Official Website", "metaDescription": "Home page for Python…",
  "contentType": "text/html", "content": "About\nNews\nDocumentation…", "contentLength": 3510, "contentTruncated": false
}

title, metaDescription, content* appear only with Fetch content. requestedDate appears with Closest snapshot to date. length is the compressed size of the archived record. A per-URL summary (snapshots returned, pages fetched, whether Max snapshots in total cut a URL short) is saved to the key-value store as SUMMARY.

Pricing

Pay per result, with lower prices on higher Apify plans, no start fee:

• Snapshot listed: $0.50 per 1,000 snapshots on the free plan, down to $0.20 per 1,000 on Gold and above.
• Page fetched (only with Fetch content, only when the archived page was actually downloaded): $2.00 per 1,000 pages on the free plan, down to $1.00 per 1,000 on Gold.

Filters, collapse and caps apply before you are charged. A run where the Wayback Machine has no captures at all fails clearly instead of silently returning an empty dataset; a filtered run with no matches succeeds with 0 results.

Limits and notes

• The Internet Archive rate-limits its APIs, so the actor stays at about one request per second with automatic back-off. Big listings can take the archive a minute or more to answer; fetching page content is roughly one page per second.
• For a few very large sites the Wayback Machine does not allow prefix/host/domain listings. Those URLs return the captures of the exact URL instead, with a note in SUMMARY.
• Large domains can have millions of captures: use collapse, filters and maxItems (default 1,000).
• Only captures the Internet Archive has made public are returned; excluded sites return nothing.
• Content is returned for text formats (HTML, XML, JSON, plain text) up to 5 MB per page; binary captures are listed but not downloaded.
• Public data from the Internet Archive's documented public APIs.