Wayback Machine Scraper: Snapshots, Changes & Page Text

Give it a list of URLs or domains and get back every capture the Internet Archive's Wayback Machine holds for them: when each page was archived, the HTTP status and type of what was captured, a content hash that shows when the page changed, and direct links to the archived copy. Optionally it also downloads each archived page and gives you its title and visible text.

One row per snapshot, ready for a spreadsheet, a database or an AI pipeline. It reads the Archive's own public index (the Wayback CDX Server API), so there is no browser, no login and no API key.

You pay per snapshot row. A URL with no captures, an address the Archive doesn't show, and a moment when the Archive is too busy to answer still get a row so you know what happened, but they cost nothing.

Who it's for

• SEO and domain research: see how a site, a page or an expired domain looked over the years, find every URL a site ever had (for redirect maps and lost content), and check what was on a domain before you buy it.
• Journalists, researchers and lawyers: prove when a page said something, and track when a policy, price list or terms page changed.
• Data and AI teams: pull the text of old versions of pages in bulk, one clean row each.
• Developers: a ready-made, polite CDX client with paging, retries and clean typed output.

What it does

• Any scope. One page (https://www.example.com/about), everything under a path (example.com/blog/*), a whole host (www.example.com) or a domain with all its subdomains (*.example.com).
• Filters the Archive applies for you: date range, HTTP status codes (200, 404, or a class like 3xx), and MIME types (text/html, application/pdf, image/*).
• Thins out long histories: keep the first capture per hour, day, month or year, only the captures whose content changed (by content hash), or one capture per distinct URL (a list of every archived URL on a site).
• Streams big histories. Results are read in pages of 1,000 with the index's own resume keys, so tens of thousands of captures per input are fine. maxSnapshotsPerInput caps each input, and the run summary says when more exist.
• Archived page text (optional). For each HTML or text capture it downloads the raw archived page (the id_ form, exactly as captured, without the Wayback toolbar), strips scripts, styles and markup, and returns the title and text, one line per block, cut at the length you choose. Images, PDFs, redirects and error captures are recognized and skipped without a charge.
• Polite to the Archive. At most one index request and two page requests per second for the whole run, answers like "slow down" (429) and "busy" (503) are waited out as the Archive asks, and every 429 makes the rest of the run slower. web.archive.org's robots.txt is obeyed for page downloads. The tool names itself in its User-Agent (WaybackSnapshotsBot).

Compared with other ways to get this

	This tool	Calling the CDX API yourself	Tools that open each Wayback page in a browser
Setup	Paste URLs, click Start	Write paging, retries and parsing code	Paste URLs
Whole paths, hosts and domains	Yes, with wildcards	Yes	Often single pages only
Date, status, type filters and collapsing	Yes, done by the Archive	Yes	Varies
Big histories	Streamed with resume keys	You handle paging	Slow: one browser page per capture
Archived page text	Optional, raw capture without the toolbar	Extra code	Usually the main feature
Cost model	Per snapshot row, text extra only when extracted	Free, your own machine and time	Often compute time per page
Empty or failed inputs	Free row with the reason	Your own handling	Varies

Input

Field	What it does
URLs or domains	One per line. A page, a domain, example.com/path/* (everything under it) or *.example.com (domain and subdomains). Repeats are looked up once. Local and private network addresses are refused.
Match	How entries without a wildcard are matched: exact (default), prefix, host or domain.
From date / To date	UTC, as 2015, 2015-06, 2015-06-30 or 20150630123000. Both ends are included, so to: 2019 runs to the end of 2019.
HTTP status codes	Keep only these, e.g. 200 or 3xx. Empty keeps everything, including "revisit" records (captures the Archive stored as "same as before", which have no status of their own).
MIME types	Keep only these, e.g. text/html or image/*.
Keep one capture per	none (default), hour, day, month, year, digest (changed content only) or url (one per distinct URL).
Max snapshots per input	Default 1,000, up to 100,000.
Get archived page text	Off by default. On: title and text of each archived HTML or text page.
Max text characters per page	Default 20,000 (500 to 200,000).

Example input:

{
  "urls": ["example.com", "https://www.nasa.gov/", "example.org/blog/*"],
  "from": "2010",
  "to": "2020",
  "statusCodes": ["200"],
  "collapse": "year",
  "maxSnapshotsPerInput": 100,
  "fetchContent": false
}

Output

One row per snapshot (values illustrative):

{
  "inputUrl": "example.com",
  "matchType": "exact",
  "status": "snapshot",
  "originalUrl": "http://www.example.com/",
  "timestamp": "2016-07-01T12:00:00Z",
  "waybackTimestamp": "20160701120000",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "OKOGNYOM6RNZIFU4BUKUYOKHUNEWXSSX",
  "length": 1255,
  "archiveUrl": "https://web.archive.org/web/20160701120000/http://www.example.com/",
  "rawUrl": "https://web.archive.org/web/20160701120000id_/http://www.example.com/",
  "title": "Example Domain",
  "text": "Example Domain\nThis domain is for use in illustrative examples in documents. ...",
  "textStatus": "ok",
  "textTruncated": false,
  "textError": null,
  "error": null,
  "scrapedAt": "2026-11-04T10:15:00.000000+00:00"
}

• timestamp is the capture time in UTC; waybackTimestamp is the same in the Archive's 14-digit form.
• digest is the Archive's hash of the captured content: the same digest means the same content.
• length is the size of the stored (compressed) record in bytes, not the page size.
• archiveUrl opens the capture in the Wayback Machine; rawUrl is the capture exactly as it was served.
• title, text, textStatus, textTruncated and textError are there only when Get archived page text is on. textStatus is ok (charged), not_text (image, PDF and the like), not_a_page (the capture was a redirect or an error page), no_text (nothing visible on the page), failed (the Archive could not serve it; textError says why) or blocked_by_robots.

status is one of:

• snapshot: one archived capture. This is the charged row.
• no_snapshots: the Archive has no captures for this input with these filters.
• excluded_from_archive: the Archive refuses to list captures for this address. Site owners can ask for that, and some large sites are only listed to signed-in Archive accounts. These rows are free.
• archive_busy or archive_error: the Archive didn't answer even after waiting and retrying (error says what happened). Try again later.
• invalid_input: the entry wasn't a usable public web address.

The run also saves a SUMMARY record: snapshots saved and charged, text extracted, inputs per status, index requests, and for each input whether more captures exist beyond your limit.

Pricing

Pay per event:

• $0.0005 per snapshot row ($0.50 per 1,000).
• $0.002 extra per snapshot whose archived page text was extracted ($2 per 1,000), only when Get archived page text is on and the text was actually extracted.

Rows for inputs with no captures, excluded or failed inputs and invalid entries are free. No start fee. Set a maximum charge per run in the run options: before each batch is read its full price (including text) is set aside, so the run stops cleanly at your limit and never goes past it.

Good to know

• Speed. The index answers in pages of up to 1,000 captures, at most one page per second, so listing captures is fast. Archived page text is about two pages per second at most (the Archive is often slower than that), so 1,000 pages of text take roughly 10 to 20 minutes.
• Huge domains. A domain-wide or prefix query on a very large site (a big news site, a social network) with no date range can take the Archive a long time to answer. Add a date range or a collapse option and keep the limit modest.
• Collapsing looks at neighbouring captures. "Changed content only" drops a capture when it is identical to the one right before it, so a page that changes and changes back shows both changes. With prefix, host or domain matching the index is ordered by URL, then time.
• Revisit records. When the Archive saw the same content again it may store a "revisit" (mimeType: warc/revisit, no status). Filtering on status 200 leaves them out; with text on, they are read from the capture they point to.
• Text is a plain extract. It's the visible text of the page as captured: menus and footers included, no layout, no JavaScript-rendered content.
• Times are UTC, as the Archive records them.

Responsible use

• It reads only the Internet Archive's public index and public archived pages, the same ones anyone can open at web.archive.org. No logins, no private data.
• It keeps its requests modest, waits when the Archive asks it to, obeys web.archive.org's robots.txt for page downloads, and identifies itself in every request.
• Archived pages are copies of other people's websites. You are responsible for how you use them: respect copyright and privacy laws that apply to you.

This Actor isn't affiliated with or endorsed by the Internet Archive. If it's useful to you, consider donating to the Internet Archive.

More tools from the same developer

• Broken Link Checker: 404s, Redirects & Dead Links
• SEO Audit Tool: On-Page Checks, Crawl & Broken Links
• Sitemap URL Extractor: All Site URLs, Lastmod & News
• Bulk URL Status Checker: Redirect Chain & Response Time
• Website Screenshot & PDF: Full Page, Mobile, Batch URLs
• Domain WHOIS, DNS & SSL Checker: Expiry, SPF & DMARC