Wayback Machine Toolkit — Archived URLs, Snapshots & Diffs

Scrape the Wayback Machine three ways from one Actor: list every archived URL for a page, path or whole domain, read what a page said on a given date, and diff one page between two dates.

Everything comes from the Internet Archive's own public CDX and replay endpoints — no API key, no login, no proxies — and a CDX query returns in seconds even for thousands of rows. Index rows cost $0.0002 each ($0.20 per 1,000), and a URL that was never archived is a free diagnostic row.

The third mode is the point. Other archive tools list snapshots or fetch them. This one answers the question people actually have — what changed on this page between January and June? — in a single row, with the added and removed lines, a similarity score, and the headings and prices that moved.

Try it in 30 seconds

Every archived capture of a page — this is the Store prefill; paste it and run:

{ "mode": "cdx", "urls": ["https://news.ycombinator.com"], "limit": 20 }

What changed on a page over a decade:

{ "mode": "diff", "urls": ["https://news.ycombinator.com"], "dates": ["earliest", "latest"], "diffGranularity": "line" }

What a page said on a given date:

{ "mode": "snapshot", "urls": ["https://news.ycombinator.com"], "dates": ["20200105"], "snapshotFormat": "markdown" }

Output — one row per archived capture in cdx mode, one per date in snapshot mode, one per comparison in diff mode; the fields you will use most are originalUrl, timestamp, isoDate, statusCode, mimeType and archiveUrl (full list under Output reference). A URL that was never archived comes back as a free diagnostic row (ok: false, errorType, error) instead of a charge.

Price — $0.20 per 1,000 index rows on the FREE tier ($0.0002 each), $1.50 per 1,000 snapshots fetched, $6 per 1,000 diff reports, plus a $0.0005 start fee charged only after the first paid row is delivered; diagnostics, collapsed duplicate captures and a run that returns no row are free. Pay-per-event, no API key, no browser, limited permissions — it works over the Apify MCP server (mcp.apify.com) and with agentic (x402) payments.

From code — client.actor("insight.solutions/wayback-toolkit").call(run_input={…}) with apify-client, or POST https://api.apify.com/v2/acts/insight.solutions~wayback-toolkit/run-sync-get-dataset-items. Full snippets under Use it from an AI agent, or from code.

Three modes: archived URL index, page snapshot, page diff

Mode	Give it	Get back	Charged as
cdx	a URL or domain, optional date range and filters	One row per archived capture: timestamp, ISO date, archived status, MIME type, digest, size, Wayback URL	snapshot-index-row
snapshot	a URL and one or more dates	One row per capture, with the archived page as text, Markdown or HTML, plus title, headings, links and word count	snapshot-fetched
diff	a URL and exactly two dates	One row: added and removed lines, similarity ratio, word counts, heading changes, price changes, summary	diff-report

A few more things worth knowing up front:

• Archived pages come back clean. Fetched through the id_ raw endpoint, so no Wayback toolbar is injected into your data.
• Only content changes are billed by default. The index collapses on content digest, so a page that sat unchanged for two years is one row, not four hundred.
• Never an empty dataset. A URL that was never archived produces a diagnostic row explaining what to try instead — and is not charged.
• Honest empty runs. A run whose input was usable but produced nothing — a URL that was never archived, a page with one capture to diff, archive.org refusing every request — finishes SUCCEEDED with zero results, a status message that says so, and a free diagnostic row telling you why. Whether archive.org never answered or answered with nothing, nothing is charged, start fee included. A run finishes FAILED only when the Actor itself hit an error; invalid input is reported in the status message, uncharged.

Use cases

• Get all archived URLs for a domain — matchType: "domain" gives you the URL inventory the archive holds, in one request.
• Compare two versions of a website — diff a competitor's pricing page between any two dates and read removedPrices / addedPrices. On a schedule, that is a price-change timeline nobody is publishing.
• SEO recovery after a site migration — matchType: "prefix" gives you the URL inventory a site had before a redesign, which is the list you need to build redirects from.
• Domain vetting before a purchase — a gap in the index, a run of 404s, or a sudden switch to a different language is how you find out a domain was a spam farm before you pay for it.
• Compliance and evidence capture — fetch what a page said on the day a claim was made, with the archive's own timestamp, digest and a citable Wayback URL on the row.
• Content-change monitoring — diff with dates: ["20240101", "latest"] on terms of service, policy pages or documentation, scheduled monthly.
• Recover lost content — snapshot in markdown turns an archived page back into something you can paste into a CMS.

How it compares

• Diff is the product, and nothing else in this category has it. Listing snapshots is easy. The hard part is that a 2015 page and a 2024 page of the same site share almost no HTML — table layouts became flexbox, class names churned, analytics tags came and went — so diffing raw archived markup returns thousands of meaningless changes and buries the one that matters.
• $0.20 per 1,000 index rows, where comparable archive listers run to several dollars per 1,000 items — and digest collapsing means you are not billed for years of stasis.
• Three tools in one Actor rather than one Actor per job, on a single input schema.

What the diff actually does:

1. Fetches raw bytes, not the replay page — always via https://web.archive.org/web/{timestamp}id_/{url}. Without the id_ suffix, archive.org injects its own toolbar and rewrites every link, and that chrome lands in your diff.
2. Decodes the archived charset. A 2003 capture is frequently windows-1252, not UTF-8. The archived Content-Type is honoured, with a <meta charset> sniff and a UTF-8 fallback, so accented text survives instead of becoming mojibake that reads as a change.
3. Reduces both captures to block-level text — one line per heading, paragraph, list item or table cell, with script, style, nav and footer removed, so whitespace and markup churn register as no change at all.
4. Compares at line and word level, with a similarity ratio of 2 × matching words / total words — the same measure Python's difflib.SequenceMatcher reports. 1 means identical text even when the HTML is not; 0 means nothing survived.
5. Overlays what people actually look for — headings that appeared or disappeared, and currency amounts that changed ($, £, €, ¥, ₹, and ISO codes either side of the number). Percentages and bare numbers are deliberately not treated as prices: a false positive in pricesChanged is worse than a miss.
6. Always compares oldest → newest, whichever order you write the two dates in, so "added" means "added over time".

Input reference

Field	Type	Default	Notes
mode	cdx | snapshot | diff	cdx	Required
urls	array of strings	—	URLs or domains. Scheme optional. A single URL may also be given as a plain url string
matchType	exact | prefix | host | domain	exact	cdx mode only. snapshot and diff always match exactly — they compare a page against itself
from / to	date	—	YYYYMMDD, an ISO date, or a partial date such as 2019 or 201907
statusFilter	array of strings	["200"]	Archived HTTP statuses to keep. Empty = all, including archived redirects and error pages
mimeFilter	array of strings	["text/html"]	Archived Content-Types to keep. Empty = all, including images and PDFs
collapse	digest | timestamp:8 | timestamp:6 | urlkey | none	digest	cdx mode. digest = one row per content change; timestamp:8 = daily; timestamp:6 = monthly
limit	integer 1–10000	200	Hard cap on index rows per URL, enforced by the archive before anything is billed
dates	array of strings	see notes	earliest, latest, or a date; the nearest capture is used. snapshot: any number, empty means the latest capture. diff: exactly two, empty means earliest vs latest
snapshotFormat	text | markdown | html	text	snapshot mode
diffGranularity	line | word	line	word additionally returns every changed word run
maxSnapshotKb	integer 16–8192	2048	Per-document read budget. Anything longer is cut and flagged truncated: true
maxRunSecs	integer 30–3600	240	Wall-clock budget. On reaching it the run stops requesting more, keeps every row written, and says so

Output reference

Every row, in every mode, carries scrapedAt, source and sourceUrl — the exact archive.org URL it was built from.

Index rows (cdx) — ok, mode, queryUrl, originalUrl, timestamp, isoDate, statusCode, mimeType, digest, length, archiveUrl, rawArchiveUrl:

{
  "ok": true,
  "originalUrl": "https://example.com/pricing",
  "timestamp": "20240620093000",
  "isoDate": "2024-06-20T09:30:00.000Z",
  "statusCode": "200", "mimeType": "text/html",
  "digest": "PRICING2024PRICING2024PRICING202", "length": 3120,
  "archiveUrl": "https://web.archive.org/web/20240620093000/https://example.com/pricing"
}

Snapshot rows (snapshot) — all of the above plus requestedDate, title, content, contentFormat, wordCount, headings[], links[], contentBytes, contentCharset, truncated, indexUrl.

Diff rows (diff) — url, fromTimestamp, toTimestamp, fromIsoDate, toIsoDate, fromArchiveUrl, toArchiveUrl, fromStatusCode, toStatusCode, fromDigest, toDigest, granularity, identical, similarityRatio, addedLines[], removedLines[], addedLineCount, removedLineCount, unchangedLineCount, changedCount, addedLinesTruncated, removedLinesTruncated, fromLineCount, toLineCount, addedWords[], removedWords[], addedWordCount, removedWordCount, fromWordCount, toWordCount, fromTitle, toTitle, titleChanged, addedHeadings[], removedHeadings[], headingsChanged, fromPrices[], toPrices[], addedPrices[], removedPrices[], pricesChanged, fromTruncated, toTruncated, summary:

{
  "fromIsoDate": "2015-06-12T09:00:00.000Z",
  "toIsoDate": "2024-06-20T09:30:00.000Z",
  "similarityRatio": 0.7379,
  "changedCount": 14,
  "addedLines": ["$15 per month", "$39 per month", "Enterprise", "…"],
  "removedLines": ["$9 per month", "$29 per month", "…"],
  "addedHeadings": ["Enterprise", "Enterprise add-ons"],
  "removedPrices": ["$9", "$29", "$99"],
  "addedPrices": ["$15", "$39", "$149"],
  "pricesChanged": true,
  "summary": "73.8% of the text is unchanged; 8 line(s) added and 6 line(s) removed; new price(s): \"$15\", \"$39\", \"$149\"; prices on this page changed."
}

Diagnostic rows — ok: false, plus error, errorType, checkedAt. Never charged.

Pricing

You pay per result, not per minute. There is no subscription and no platform-usage surcharge.

Event	What triggers it	Free	Bronze	Silver	Gold
actor-start	Once per run, only after the first paid row is delivered	$0.0005	$0.0005	$0.0005	$0.0005
snapshot-index-row	Each archive index row returned	$0.0002	$0.0002	$0.0002	$0.0002
snapshot-fetched	Each archived page fetched and parsed	$0.0015	$0.0015	$0.0012	$0.0009
diff-report	Each two-date comparison returned	$0.006	$0.006	$0.0048	$0.0036

Bronze is the Starter plan, Silver is Scale, Gold is Business. Higher plans pay less per result.

What that costs in practice

You ask for	You get	You pay
Every capture of one page, limit: 20	20 index rows	$0.0005 + 20 × $0.0002 = $0.0045
Every archived URL under a domain, limit: 5000	5,000 index rows	$0.0005 + 5,000 × $0.0002 = $1.00
One archived page read at three dates	3 snapshots	$0.0005 + 3 × $0.0015 = $0.005
A competitor's pricing page compared across 10 years	1 diff report	$0.0005 + $0.006 = $0.0065
50 competitor pages diffed year-over-year	50 diff reports	$0.0005 + 50 × $0.006 = $0.30

You are never charged for: diagnostic rows — a URL that was never archived, a filter matching no captures, a page with only one capture to diff; anything at all on a run that returns no paid row — the actor-start fee is billed once, and only right after the first index row, snapshot or diff report has been delivered to your dataset, so an archive.org outage, a URL that was never archived or a page with one capture costs nothing and finishes SUCCEEDED with zero results and a status message ending Nothing was charged.; captures the index collapsed as unchanged; rows the Actor could not write because your run or time budget was exhausted; anything beyond the four events above — no platform-usage pass-through.

Performance notes

archive.org is slow, and it is a donation-funded public archive. This Actor treats it accordingly: at most two concurrent requests with a minimum gap between request starts (not configurable upward); Retry-After obeyed exactly on 429 and 503, otherwise exponential backoff with jitter, up to three retries; a 45-second per-request timeout, because a cold capture can take that long to come out of storage; and a per-document read budget (maxSnapshotKb, default 2 MB) so an enormous archived document is truncated with truncated: true rather than killing the run.

Index first, then fetch only the captures you want — a cdx query is one request per URL and returns in seconds even for thousands of rows, while a snapshot costs one index request plus one fetch per date and a diff costs one index request plus two fetches. A matchType: "domain" query on a large site can match millions of captures, and limit is a hard cap applied by the archive, so raise it deliberately. Batching many URLs into one run beats many single-URL runs — but raise maxRunSecs when you do, since the default 240 s is sized for a handful of URLs, not a hundred.

Use it from an AI agent, or from code

One JSON object in, one flat array out — the shape agent runtimes want. The Actor runs with limited permissions, uses pay-per-event pricing and never enters Standby, so it works over the Apify MCP server and with x402 agentic payments. The Integrations tab pushes results to Slack, a webhook, Zapier, Make, Google Sheets, Snowflake or BigQuery, and Apify Schedules run it unattended — a monthly diff on a competitor's pricing page with dates: ["20240101", "latest"] is the common pattern.

curl -X POST "https://api.apify.com/v2/acts/insight.solutions~wayback-toolkit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"diff","urls":["example.com/pricing"],"dates":["earliest","latest"]}'

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("insight.solutions/wayback-toolkit").call(run_input={
    "mode": "diff",
    "urls": ["example.com/pricing", "example.com/terms"],
    "dates": ["20230601", "20240601"],
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row.get("ok"):
        print(row["url"], row["similarityRatio"], row["addedPrices"], row["summary"], sep=" | ")

FAQ

Why is a capture missing that I can see in the Wayback Machine? Almost always the default filters. statusFilter: ["200"] drops archived redirects and error pages, and mimeFilter: ["text/html"] drops everything that is not a page. Empty both to see the raw index. collapse: "digest" also hides captures byte-identical to the one before — set it to none to see every crawl.

Why do http:// and https:// versions show up as one list? The archive indexes both under a single canonical key, so a query for example.com returns captures of both. The originalUrl field on each row tells you which it was.

What does similarityRatio actually measure? 2 × matching words / total words across both captures, over the extracted visible text. It ignores HTML entirely. 1 means identical text; 0 means nothing shared.

Why does a diff show changes when the page looks the same? Check addedLines and removedLines. Common culprits are dates, view counters, "last updated" strings and cookie banners — real text changes a human eye skips over. Anything inside script, style, nav and footer is already excluded.

Can it diff more than two dates? One diff row compares exactly two captures. For a timeline, run snapshot across many dates and compare the rows yourself, or run several diffs with different date pairs in one run.

What if a page was only archived once? The run finishes successfully with a diagnostic row saying so, and you are not charged for the attempt.

How fresh is the archive? That is the Internet Archive's crawl schedule, not ours. A popular page may be captured daily; an obscure one may have a five-year gap. cdx mode shows you the real cadence for any URL.

Legal

• Not affiliated with, endorsed by, or sponsored by the Internet Archive. "Wayback Machine" and "Internet Archive" are used descriptively to identify the public data source this Actor reads from.
• All data comes from the Internet Archive's public, unauthenticated endpoints, which exist to be queried programmatically. No login, no authentication, no paywall is bypassed, and no content is fetched from the live web.
• Archived pages remain the copyright of their original publishers. This Actor retrieves and compares them; what you do with the text is your responsibility, and normal copyright and fair-use rules apply.
• The Actor holds itself to at most two concurrent requests with backoff and Retry-After compliance, so that using it does not degrade a free public service for anyone else.
• Archived pages can contain personal data that their publishers have since removed. This Actor does not extract, enrich, or infer personal data; it returns page text as archived. If you are processing personal data out of it, that is your obligation as controller, not ours.

Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

Video, audio & social

• YouTube Transcript API — captions as timed segments, text, SRT or VTT, with language fallback and translation.
• YouTube Comments API — comments and replies with likes, pinned and hearted flags, newest or top sort.
• YouTube Channel API — a channel's videos, Shorts and live streams, plus YouTube search.
• Podcast Search, Episodes & Charts API — Apple Podcasts search, charts and full episode feeds.
• Bluesky Scraper — profiles, posts, followers and follows from the public AT Protocol API.
• Telegram Channel Scraper — posts, views and channel stats from public Telegram channels.
• Substack Scraper — posts with full free text, comments and publication profiles.
• Hacker News API — stories, comments, users, front page and a structured "Who is hiring?" parser from the official HN APIs.
• Discourse Forum API — topics, posts and categories from any Discourse community via its own JSON endpoints, usernames only.

News, documents & the web

• Google News Search, Topics & Real Article URLs — news search and topic feeds with the publisher's real URL decoded.
• Website to Markdown — Content Extractor for LLMs & RAG — any site as clean Markdown, text and heading-aware chunks.
• Internet Archive API — archive.org search, item metadata, files and reviews.
• Website Technology Detector — the tech stack behind any site, with the evidence for each detection.
• Domain Intelligence API — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
• SEO Page Audit — sitemap crawl with on-page checks, structured data and broken-link reports.
• Keyword Suggestions API — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
• Website Contact Extractor — emails, phone numbers and social profiles from any list of websites.
• Web Search Results API — Bing and DuckDuckGo organic results with snippets, no key, no browser.
• Company Enrichment API — a domain in, a company profile out: firmographics, contacts, tech stack, DNS and hiring signal.
• Company Dossier API — one company in, twelve sections out: profile, tech, contacts, DNS, open roles, news, SEC filings, federal awards, recalls, YC batch and apps.
• Press Releases API — GlobeNewswire and PR Newswire releases plus any newsroom feed, by keyword, company, ticker or subject.
• Federal Register API — rules, proposed rules, notices and the Public Inspection desk with dockets, comment deadlines and CFR references.
• Academic Papers Search API — OpenAlex, Crossref, arXiv and PubMed in one row per paper: abstract, citations, open-access PDF, authors and venue.
• RSS & Atom Feed Monitor — any RSS, Atom or JSON feed (or an OPML file) in, only the new items out, with keyword filters and a webhook.
• Website Change Monitor — watch any pages, diff the text between runs, get change rows with added/removed lines, keyword alerts and a webhook.
• Wikipedia & Wikidata API — article text, search, daily pageviews and Wikidata entity facts, any language edition.

Business, finance & jobs

• Congress & Insider Trades API — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
• Federal Contracts, Grants & Lobbying API — SAM.gov opportunities, USAspending awards, Grants.gov notices and Senate lobbying filings in one schema.
• SEC EDGAR API — filings, XBRL financials and full-text search by ticker or CIK.
• Clinical Trials & FDA API — ClinicalTrials.gov studies plus openFDA recalls, labels, approvals, 510(k)s and adverse-event reports.
• Product & Vehicle Recalls API — CPSC, NHTSA, FDA and USDA recalls, vehicle complaints and ratings, plus a VIN decoder.
• Y Combinator Companies, Batches & Founders — the YC directory with founders and social links, filterable by batch, industry and hiring status.
• Career Site Jobs API — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
• New Job Postings Monitor — new, closed and changed postings on the career sites you watch.
• Hiring Signals API — Open Roles & Hiring Surge by Company — one row per company per run: open roles, what opened and closed, department and seniority breakdowns, and a hiring-surge flag.
• Remote Jobs API — RemoteOK, Remotive, We Work Remotely, Himalayas, Jobicy and more in one schema, deduplicated.
• Shopify Products API — any Shopify store's catalogue, variants, prices and stock signals.
• Shopify Store Monitor — price drops, sales, restocks, sell-outs and new products on any Shopify store, one row per change.
• Public Tenders API — EU TED, UK Find a Tender and Contracts Finder notices by keyword, CPV code, country, stage and deadline.
• Nonprofit & IRS 990 Lookup API — search US nonprofits and get EIN, NTEE code and multi-year Form 990 financials.
• OpenStreetMap Places API — businesses and points of interest by category and area from OpenStreetMap: name, address, coordinates, website, phone, opening hours.

Apps & games

• App Store & Google Play Reviews API — reviews from both stores with ratings, versions and developer replies.
• App Store Top Charts & App Search API — Apple top charts by country and genre, plus app search and details.
• App Store Keyword Rank Tracker — where any app ranks for any keyword on the App Store and Google Play, with rank changes and ASO suggestions.
• Steam Reviews API — Steam reviews with playtime, helpfulness and game details.
• Steam Game Data API — prices, tags, review scores, live player counts and top charts.