Wayback Machine Scraper - Archived URLs & Website History

What does Wayback Machine Scraper do?

It reads the Internet Archive's Wayback Machine for any list of websites or pages and returns clean rows you can filter, export or feed to other tools. Four modes:

• 🗂️ Archived URLs of a site: every page the Wayback Machine has ever captured for a domain, a host or a path (e.g. example.com/blog/), with the first capture date and snapshot links. Find deleted pages, old URLs for redirect maps after a site migration, or content to recover
• 🕰️ Snapshots of a page over time: every archived version of one page. By default only captures where the content changed (one row per version), or at most one per day, month or year
• 🎯 Closest snapshot per URL: the snapshot nearest to a date, or the latest one. Paste a list of dead links and get a working archive link for each
• 📈 Site history summary: one row per domain or page with first and last capture, the last month (and day) with a real page (HTTP 200), months that only redirected, captures per year, number of content versions and the longest gap without captures. Useful before buying an expired domain or when researching a company's past
• 📝 Optional page content: in Snapshots and Closest modes, each archived page can be downloaded and returned as clean Markdown (or plain text), ready for an LLM, a diff or a report
• 🧹 Clean, de-duplicated rows: http://example.com:80/a/ and https://www.example.com/a count once; malformed index entries are skipped; the same fields on every row
• 🤝 Polite by design: all requests share one rate limit (default one index query every 2.5 seconds, page downloads every 5 seconds), with automatic back-off when the Internet Archive is busy
• 💵 $1 per 1,000 rows ($0.001 per archived URL or snapshot). No start fee. Inputs that were never archived are free

Who is it for?

• SEO and site migrations: list every URL a site ever had, find pages that lost their redirects, rebuild redirect maps, recover old content
• Expired and aged domain buyers: check how long a domain has been live, when it last served a real page, how often it changed and whether it spent years redirecting
• Link maintenance: turn dead outbound links into archive links (Closest mode), e.g. together with a broken-link checker
• Research, journalism, competitive intelligence: see how a pricing page, a policy page or a product page changed over the years, with the text of each version
• AI agents: "what did this page say in 2019?" answered with one call

How much does it cost?

What	Price per event	Example
Archived URL or snapshot (one row, any mode except History)	$0.001	1,000 archived URLs of a site = $1
Site history summary (History mode, one per input)	$0.003	100 domains checked = $0.30
Archived page content (optional, Markdown or text)	$0.003	20 old versions of a page with their text = $0.02 + $0.06

Larger Apify plans pay less (SILVER −10%, GOLD and above −20%):

Event	FREE and BRONZE	SILVER	GOLD and above
Archived URL or snapshot	$0.001	$0.0009	$0.0008
Site history summary	$0.003	$0.0027	$0.0024
Archived page content	$0.003	$0.0027	$0.0024

No start fee. You also pay nothing for:

• inputs that were never archived (error row not_found),
• invalid lines, duplicates and captures removed by your filters,
• page content that could not be downloaded or had no readable text.

Real example from our tests: 2 pages, closest snapshot to June 2015 plus their Markdown content, and 1 never-archived domain: 2 captures + 2 page contents = $0.008 at current prices; the run took 2 minutes and used $0.0016 of Apify platform time.

Control your cost

• Max results per input (default 1,000) and Max results (default 10,000 for the run) cap the charged rows.
• Max pages with content (default 20) caps the optional content downloads.
• At the start the run logs its worst-case cost (for example Plan: 50 captures × $0.001 = at most $0.05).
• Apify's maximum charge per run is respected: when it is reached, the run stops cleanly with status LIMIT_REACHED, and SUMMARY.notProcessed lists the inputs that were not processed, so you can raise the limit and run them.

How to use it

1. Paste websites or pages into Websites or pages, one per line (example.com, https://example.com/pricing).
2. Choose What to get: Archived URLs, Snapshots, Closest or History.
3. Optional: dates (2019, 2019-05, 2019-05-07 or 2 years), only successful captures (HTTP 200, on by default), only HTML pages, a URL regex, or page content.
4. Run. Results are in the dataset (views: Archived URLs, Snapshots / closest captures, Site history); the run summary is in the key-value store record SUMMARY.

Input examples

All archived pages of a site's blog since 2020:

{
  "urls": ["https://blog.example.com/"],
  "mode": "urls",
  "matchType": "prefix",
  "fromDate": "2020",
  "maxResultsPerInput": 5000
}

Every version of a pricing page, with the text of each version:

{
  "urls": ["stripe.com/pricing"],
  "mode": "snapshots",
  "collapse": "changes",
  "includeContent": true,
  "maxContentPages": 20
}

Archive links for a list of dead links (latest snapshot of each):

{
  "urls": ["https://www.nasa.gov/missions/", "https://example.com/old-page"],
  "mode": "closest"
}

History of domains you are thinking of buying:

{ "urls": ["pets.com", "some-expired-domain.com"], "mode": "history" }

Output example: archived URL (Archived URLs mode)

{
  "type": "url",
  "input": "python.org",
  "success": true,
  "url": "http://www.python.org/1.5/",
  "host": "www.python.org",
  "path": "/1.5/",
  "extension": null,
  "firstCapturedAt": "1998-01-19T01:50:21Z",
  "statusCode": 200,
  "mimeType": "text/html",
  "firstSnapshotUrl": "https://web.archive.org/web/19980119015021/http://www.python.org:80/1.5/",
  "latestSnapshotUrl": "https://web.archive.org/web/http://www.python.org/1.5/",
  "timestamp": "19980119015021",
  "charged": true
}

latestSnapshotUrl opens the most recent capture of that URL in the Wayback Machine.

Output example: snapshot (Snapshots mode, one row per content change)

{
  "type": "snapshot",
  "input": "stripe.com/pricing",
  "success": true,
  "target": "stripe.com/pricing",
  "index": 2,
  "url": "https://stripe.com/pricing",
  "timestamp": "20200104165645",
  "capturedAt": "2020-01-04T16:56:45Z",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "GMSUHBJQMSMCD7CWRFAY4X2OPZGY5BJW",
  "lengthBytes": 69301,
  "snapshotUrl": "https://web.archive.org/web/20200104165645/https://stripe.com/pricing",
  "rawUrl": "https://web.archive.org/web/20200104165645id_/https://stripe.com/pricing",
  "contentChanged": true,
  "daysSincePrevious": 243,
  "charged": true
}

rawUrl is the archived file exactly as captured (no Wayback toolbar). digest is the capture's content fingerprint: same digest = same content.

Output example: closest snapshot with page content

{
  "type": "closest",
  "input": "https://www.nasa.gov/missions/",
  "success": true,
  "target": "www.nasa.gov/missions/",
  "targetDate": "2015-06-01T00:00:00Z",
  "url": "http://www.nasa.gov/missions/",
  "capturedAt": "2015-06-12T18:50:45Z",
  "statusCode": 200,
  "snapshotUrl": "https://web.archive.org/web/20150612185045/http://www.nasa.gov:80/missions/",
  "daysFromTarget": 12,
  "contentTitle": "NASA Missions A-Z | NASA",
  "content": "…Markdown of the archived page…",
  "contentFormat": "markdown",
  "contentChars": 347,
  "contentTruncated": false,
  "contentCharged": true,
  "charged": true
}

Output example: site history (History mode)

{
  "type": "history",
  "input": "pets.com",
  "success": true,
  "target": "pets.com",
  "firstCapturedAt": "1996-12-21T00:37:03Z",
  "lastCapturedAt": "2026-09-20T01:06:47Z",
  "firstOkMonth": "1996-12",
  "lastOkMonth": "2016-04",
  "lastOkCapturedAt": "2016-04-14T08:28:29Z",
  "archiveAgeYears": 29.7,
  "daysSinceLastCapture": 10,
  "totalCaptures": 2391,
  "capturesByYear": { "1996": 1, "2001": 627, "…": "…", "2026": 30 },
  "yearsWithCaptures": 31,
  "monthsWithCaptures": 268,
  "monthsWithOkPages": 40,
  "monthsRedirectOnly": 228,
  "longestGapMonths": 19,
  "longestGapFromMonth": "1997-04",
  "longestGapToMonth": "1998-12",
  "daysWithCaptures": 1366,
  "distinctVersions": 117,
  "contentChanges": 95,
  "redirectShare": 0.494,
  "statusCodes": { "200": 142, "301": 449, "302": 226, "403": 4, "other": 545 },
  "dayLevelDetail": true,
  "calendarUrl": "https://web.archive.org/web/*/pets.com",
  "charged": true
}

How to read it: pets.com has captures for almost 30 years, but the last capture with a real page (HTTP 200) was in April 2016; since then it has only redirected (228 of 268 months with captures are redirect-only). Month-level fields come from the Wayback Machine's calendar data and work for any site (geocities.com: 916,869 captures, read in one request). Day-level fields (daysWithCaptures, distinctVersions, contentChanges, statusCodes, lastOkCapturedAt) come from the capture index; for extremely archived pages the index may be too slow, and the row then says dayLevelDetail: false with a note (still charged once, the month-level summary is complete).

Errors (not charged)

Every input that fails gets one row with success: false, an errorType and an error that ends with "(not charged)":

{
  "type": "capture",
  "input": "never-archived-zzq.dev",
  "target": "never-archived-zzq.dev",
  "success": false,
  "errorType": "not_found",
  "error": "never-archived-zzq.dev was never archived with HTTP 200 by the Wayback Machine (not charged)",
  "charged": false
}

Error types: not_found (never archived, or nothing in your dates or filters), invalid_input, duplicate (Archived URLs mode: every URL of this input was already listed under another input), http_error / timeout / network (the Internet Archive was busy, rate-limiting or offline after all retries: try again later), blocked (the site is excluded from the archive, HTTP 403).

Integrations

Use the dataset from the Apify API, Make, Zapier, n8n, Google Sheets or webhooks. A common chain: a broken-link checker finds dead links → this Actor's Closest mode returns an archive link for each → your CMS or a sheet gets the replacement links.

Use with AI agents (MCP)

Connect Apify's MCP server (https://mcp.apify.com?tools=tidytools/wayback-machine-scraper) to Claude, Cursor or any MCP client, then ask e.g. "What did stripe.com/pricing say in January 2020? Give me the text." or "When did pets.com last serve a real page?"

Minimal input:

{ "urls": ["stripe.com/pricing"], "mode": "closest", "targetDate": "2020-01", "includeContent": true }

Failed items are not charged and carry an errorType. The run summary (SUMMARY) has a status field: SUCCESS, PARTIAL_RESULTS, NO_RESULTS, FAILED or LIMIT_REACHED.

Tips and limitations

• The Wayback Machine is a shared, free public service and can be slow. Index queries for heavily archived pages can take 30–60 seconds, and the service is sometimes briefly offline. The Actor waits and retries (not charged). In History mode, very heavily archived pages may get the month-level summary only (dayLevelDetail: false).
• Archived URLs mode returns the first capture of each URL. Use latestSnapshotUrl, or Snapshots / Closest mode, for later versions.
• In History mode, totalCaptures and capturesByYear count every capture; daysWithCaptures counts days with at least one capture.
• Only what the archive has: pages that were never crawled, blocked by the site owner or removed on request are not available.
• Page content is the archived HTML converted to Markdown. Pages that relied on JavaScript to show their text may have little content (contentChars).
• Default rate: one index query every 2.5 seconds (the index starts refusing at about 24 queries a minute), page downloads every 5 seconds. You can slow it down in Advanced settings; faster than 2 requests per second is not allowed.

Terms and responsible use

This Actor only reads the Internet Archive's public Wayback Machine endpoints (the CDX index server, the availability API and snapshot playback), at a gentle rate, and does not log in anywhere. Archived pages remain the work of their original owners: check copyright before republishing their content, and follow the Internet Archive's terms of use. The Actor does not collect personal data on purpose; if archived pages contain personal data, you are responsible for handling it lawfully (e.g. GDPR). We are not affiliated with the Internet Archive; if you use the archive heavily, consider donating to it.

FAQ

Does the Actor collect usage data? The Actor sends one anonymous usage counter per run (input type, item count range, outcome) to improve the tool; no URLs, content or account data.

Is there an official API? Yes, and this Actor uses it: the Wayback Machine's public CDX server and availability API. The Actor adds de-duplication, date parsing, history summaries, content conversion, retries, cost limits and ready-made exports.

Why do I get fewer URLs than the Wayback Machine website shows? By default only successful HTML pages are listed (no images, scripts, redirects or errors), and scheme, www and trailing-slash variants are merged. Turn off "Only successful captures" and "only HTML pages" to see everything.

Can I paste a web.archive.org link? Yes. A link such as https://web.archive.org/web/20200101000000/https://example.com/pricing is read as the page it archived (https://example.com/pricing); several links to the same page count once.

What if a date, the URL pattern or the list is wrong? Nothing is read and nothing is charged: you get one free invalid_input row that says what to fix (for example 2019-13, 2012-06-31 (June has 30 days), a "From date" in the future or after the "To date", a "Target date" in the future (leave it empty for the latest capture), or an empty list) and the run ends normally. A single bad line in a longer list gets its own free row and the other inputs still run.

What if two of my inputs overlap (for example example.com/ and example.com/blog/ with Match = prefix)? In Archived URLs mode each URL is listed and charged once per run, under the first input that returned it; the other input says how many were already listed. Query strings are part of a page address (example.com/?a=1 is a different page from example.com/), and #fragments are ignored.

Does it respect robots.txt? The Actor never visits the archived sites themselves; it only reads the Internet Archive's own index and playback servers, which decide what is archived and what is excluded. Pages the site owner had excluded return an error row (blocked, not charged).

What if the Internet Archive is slow, rate-limiting or down? The Actor waits and retries with growing pauses (up to about 45 seconds). If it still cannot get an answer, that input gets one free error row (http_error or timeout) and the other inputs continue. For very large sites in Archived URLs mode, narrow the dates, add a URL pattern or use a longer path with Match = prefix.

Can it monitor a page for changes? Snapshots mode with a fromDate shows new versions since that date. For live monitoring of the current page, use a change-monitoring Actor.

Can I get the exact old file (HTML, PDF, image)? Yes: every row has rawUrl, the file exactly as captured.

Can I export every archived URL of a domain from archive.org? Yes. Set What to get to Archived URLs of a site and enter a domain, a host or a path (e.g. example.com/blog/): you get every page the Wayback Machine has captured, with the first capture date and snapshot links, at $1 per 1,000 rows. Inputs that were never archived are free.

Found this useful? A short review on the Store helps other users find it.

Related Actors

More tools by TidyTools that work well with this one:

• Website Change Monitor: get an alert when a page, a price or a text changes
• Bulk URL Status Checker: check HTTP status and redirects for a list of URLs
• Broken Link Checker: find 404 and dead links on any website
• Bulk WHOIS & DNS Lookup: domain age, expiry, DNS and email security for many domains