# Wayback Machine Scraper: Snapshots, Page Text & Domain History (`varnmill/wayback-machine-scraper`) Actor Internet Archive Wayback Machine data without login: snapshot lists (CDX) with date, status and MIME filters, archived pages as clean text, Markdown or HTML for any date, domain history reports for SEO and expired domains, and archive.org item search. Pay per result, from $0.50 per 1,000. - **URL**: https://apify.com/varnmill/wayback-machine-scraper.md - **Developed by:** [Jay Novello](https://apify.com/varnmill) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.35 / 1,000 snapshots This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Wayback Machine Scraper: Snapshots, Page Text & Domain History Everything the Internet Archive's Wayback Machine holds about a URL or domain, as JSON, CSV or Excel: every capture with date, status and size, the archived page itself as clean text, Markdown or raw HTML for any date, a domain history report (first and last seen, captures per year, homepage titles over time, redirects, parked periods) and archive.org item search; no login, no API key, no browser. **Try it:** Click Start with the default input: 100 snapshots of apify.com (one per month, 2007 to 2024) in 10 s for about $0.05 (run on October 21, 2026). Five of those 100 rows (each row also has the 14-digit timestamp, content digest and a raw `id_` link without the Wayback toolbar): | Captured | URL as archived | Status | MIME | Bytes | Archive link | |---|---|---|---|---|---| | 2007-05-31 10:15 | http://www.apify.com:80/ | 200 | text/html | 1,013 | web.archive.org/web/20070531101538/http://www.apify.com:80/ | | 2014-05-17 01:14 | http://apify.com/ | 200 | text/html | 580 | web.archive.org/web/20140517011437/http://apify.com/ | | 2020-06-02 12:52 | https://apify.com/ | 200 | text/html | 32,641 | web.archive.org/web/20200602125235/https://apify.com/ | | 2023-10-02 05:29 | https://www.apify.com/ | 301 | text/html | 427 | web.archive.org/web/20231002052924/https://www.apify.com/ | | 2024-07-03 00:58 | https://apify.com/ | 200 | text/html | 48,388 | web.archive.org/web/20240703005823/https://apify.com/ | **Pricing:** $0.50 per 1,000 snapshots; archived page content $1.50 per 1,000 pages; domain history reports $6 per 1,000 domains; archive.org items $0.50 per 1,000; plus $0.00005 per run. Platform usage is included; failed runs, error rows and URLs with no capture are not charged. It uses only the Internet Archive's free public endpoints and paces itself the way archive.org asks, so large jobs are deliberately not fast (about one request per second per host). ### Real example: what was parse.com before and after Parse? Domain history mode, input `{"mode": "domainHistory", "urls": ["parse.com"], "maxTitleSamples": 20}`, cloud run of **October 13, 2026 (UTC)**. Excerpt of the one dataset row: ```json { "domain": "parse.com", "firstSeen": "1996-11-03T04:51:13Z", "lastSeen": "2026-10-10T18:49:03Z", "ageYears": 29.9, "homepageCaptures": 9814, "yearsWithCaptures": 30, "missingYears": [2010], "longestGapMonths": 25, "redirectTargets": ["parseplatform.org"], "parkedYears": ["2008"], "lastLiveTitle": "Parse", "lastLiveCapture": "2017-01-15T04:51:07Z", "summary": "Archived 1996-11-03 to 2026-10-10; 9814 homepage captures in 30 years; no captures in 1 year(s) (2010); 9 distinct homepage title(s); homepage redirected to parseplatform.org in 6 sampled year(s) (2018-2026); parked, for-sale or placeholder page in 2008." } ``` The `distinctTitles` field of the same row tells the story in one glance: | Years seen | Homepage title | |---|---| | 1996 | PARSE Software Devices | | 1998–1999 | PARSE Software Devices, QNX and Real-Time Consulting | | 2001–2002 | PARSE Software Devices -- Training, Consulting and Books for the QNX and Neutrino Marketplace | | 2004–2005 | PARSE Software Devices -- QNX Books, QNX Training, QNX Consulting, QNX Software, and more! | | 2007 | QNX Books, QNX Training, QNX Consulting, ... by PARSE Software Devices! | | **2008** | **Domain Name for Sale** (flagged in `parkedYears`) | | 2011–2017 | Parse | | 2018–2020 | Parse + Open Source = ❤️ (served by parseplatform.org after a redirect) | | 2021–2024 | Parse Platform | A QNX consultancy for 12 years, a for-sale page, the Facebook-owned Parse backend, then a redirect to the open-source project. That is exactly what you want to know before you buy an expired domain or judge a backlink source. The row also has `titleHistory` (one sampled homepage per year with status, final URL, redirect target, language, word count and archive link) and `capturesPerYear`. ### What it does | Mode | You give | You get | Typical use | |---|---|---|---| | **Snapshot list** (`snapshots`) | URLs, folders, hosts or domains | One row per capture: time, URL, HTTP status, MIME type, content digest, size, replay link and raw link | Find every archived URL of a site, recover deleted pages, build redirect maps, check when a page changed | | **Archived page content** (`pages`) | URLs plus dates (`2018-06-01`, `2019`, `first`, `latest`, `every-year`) or Wayback links | The capture closest to each date: title, meta description, H1s, language, canonical, clean text, Markdown, raw HTML, links | Feed old pages to an LLM, restore lost content, compare a competitor's homepage year by year | | **Domain history** (`domainHistory`) | Domains | First and last seen, age, captures per year, gaps, homepage title per year, redirects to other sites, parked or for-sale signals, optional count of archived URLs | Expired-domain vetting, SEO audits, M\&A and brand due diligence, spotting spammy past owners | | **archive.org search** (`archiveSearch`) | A query, media type, collection | Item metadata (title, creator, date, downloads, collections, subjects) and optionally every file with a direct download URL | Research datasets, public-domain books and films, software archives | #### More real output (October 13, 2026 runs) Snapshot list, input `{"urls": ["apify.com"], "collapse": "month", "maxSnapshotsPerUrl": 100}`: 100 rows in 21 seconds, the first one: ```json { "query": "apify.com", "originalUrl": "http://www.apify.com:80/", "timestamp": "20070531101538", "capturedAt": "2007-05-31T10:15:38Z", "statusCode": 200, "mimeType": "text/html", "digest": "EE6FCHP3MKBC3EV5D5Q4WQJNZNVUTNU6", "lengthBytes": 1013, "archiveUrl": "https://web.archive.org/web/20070531101538/http://www.apify.com:80/", "rawUrl": "https://web.archive.org/web/20070531101538id_/http://www.apify.com:80/" } ``` Archived page content, input `{"urls": ["apify.com"], "dates": ["2018-06-01", "first"]}` (shortened): ```json { "url": "apify.com", "requestedDate": "2018-06-01T00:00:00Z", "snapshotTimestamp": "20180531124559", "daysFromRequested": -0.5, "originalUrl": "https://www.apify.com/", "statusCode": 200, "title": "Apify - The web scraping and automation platform", "metaDescription": "Apify extracts data from websites, crawls lists of URLs and automates workflows on the web. Turn any website into an API in a few minutes!", "h1": ["The web scraping and automation platform"], "internalLinkCount": 28, "externalLinkCount": 16, "wordCount": 649, "text": "The web scraping\nand automation platform\nThe easiest way to extract structured data from any website run headless Chrome / Puppeteer jobs in the cloud ...", "parkedSignal": null } ``` The `first` row of the same run found apify.com's oldest capture, from May 31, 2007: an IIS "Under Construction" page, flagged with `"parkedSignal": "Under Construction"`. ### Pricing Pay per result. Platform usage is included, so this is the whole bill. | Event | Price | Per 1,000 | |---|---|---| | Snapshot (one CDX capture row) | $0.0005 | **$0.50** | | Archived page (text / Markdown / HTML of one capture) | $0.0015 | **$1.50** | | Domain history report (one domain, up to 40 sampled homepages) | $0.006 | **$6.00** | | archive.org item (with file list if enabled) | $0.0005 | **$0.50** | | Actor start | $0.00005 | per run | Not charged: URLs with no capture, never-archived domains, error rows and duplicate captures (two requested dates that resolve to the same snapshot). Set a maximum charge per run in Apify and the actor stops cleanly when it is reached. ### Input examples Every archived URL of a site (one row per distinct URL, HTML pages that answered 200): ```json { "mode": "snapshots", "urls": ["example.com"], "matchType": "domain", "collapse": "urlkey", "statusCodes": ["200"], "mimeTypes": ["text/html"], "maxSnapshotsPerUrl": 50000 } ``` Only the captures where a page's content changed, in 2023 and 2024: ```json { "mode": "snapshots", "urls": ["https://example.com/pricing"], "collapse": "digest", "dateFrom": "2023", "dateTo": "2024" } ``` The 20 most recent captures: `{"mode": "snapshots", "urls": ["example.com"], "newestFirst": true, "maxSnapshotsPerUrl": 20}` A competitor's homepage once a year as Markdown: ```json { "mode": "pages", "urls": ["competitor.com"], "dates": ["every-year"], "dateFrom": "2015", "contentFormats": ["markdown"] } ``` Vet a list of expired domains fast (capture statistics only, about a second each): ```json { "mode": "domainHistory", "urls": ["domain1.com", "domain2.net"], "maxTitleSamples": 0 } ``` Public-domain NASA films, most downloaded first, with download links: ```json { "mode": "archiveSearch", "searchQuery": "apollo 11", "mediaType": "movies", "collection": "nasa", "searchSort": "downloads", "maxItems": 50, "includeFileList": true } ``` ### Filters and options - **matchType**: `exact`, `prefix` (a folder), `host`, or `domain` (domain plus all subdomains). - **collapse**: one capture per hour, day, month or year; one per distinct URL (`urlkey`); or only when content changed (`digest`, which compares each capture with the previous one). - **statusCodes**: `200`, `404`, `301`, or classes such as `3xx`. **mimeTypes**: `text/html`, `application/pdf`, `image/*`. - **extraFilters**: raw CDX filter expressions for power users, for example `!original:.*\.(css|js)$`. - **dates** (pages mode): `YYYY`, `YYYY-MM`, `YYYY-MM-DD`, 14-digit timestamps, `first`, `latest`, `every-year`. You get the capture closest to the moment you ask for, and `daysFromRequested` says how far away it was. - **contentFormats**: `text` (main content, menus and footers removed), `markdown`, `html` (the original bytes as captured, no Wayback toolbar). **includeLinks** adds every link with anchor text, internal/external and rel (for example nofollow). - **maxTitleSamples** (domain history): how many archived homepages to open, one per year spread over the domain's life. Each takes 2 to 4 seconds; 0 gives capture statistics only. - **uniqueUrlsUpTo** (domain history): also count distinct archived URLs up to a cap. ### How it works, accuracy and limits - **Sources**: the Wayback CDX server (`web.archive.org/cdx/search/cdx`) for snapshot lists, `id_` raw snapshot URLs for page content, the Wayback calendar's capture summary for domain statistics, and `archive.org/advancedsearch.php` plus `/metadata` for item search. - **Politeness**: by default at most one request per second per Internet Archive host, with automatic back-off on 429 and 5xx answers (honouring `Retry-After`). If archive.org keeps answering 429 the actor slows itself further. Runs are slower than a naive scraper on purpose; it keeps your runs, and everyone else's, from getting blocked. - **The CDX server is slow on big queries.** A query can take 10 to 30 seconds. Domain-wide queries on huge sites (google.com, digg.com) can time out at the archive's side; the actor stops retrying after a few timeouts and returns an error row that suggests a narrower query (date range, collapse, or a folder prefix) instead of burning your time. - **Domain history counts are homepage captures**, not every URL on the domain. Homepage titles are sampled (one capture per year, the busiest month), so a title that lived for a few months between samples can be missed. Raise `maxTitleSamples` for more detail. - **Parked and for-sale detection** is phrase based ("domain for sale", "buy this domain", parking services, default server pages; weaker phrases like "coming soon" only count on near-empty pages). Treat it as a signal to look, not a verdict. - **Redirects**: a homepage that redirected to another registrable domain is reported in `redirectedTo` / `redirectTargets`. Redirects to its own subdomains are not. - **Text extraction** removes navigation and boilerplate. On link-heavy pages (old portals, search homepages) the main text can be mostly menu items; use `html` if you need everything. - Sites excluded from the Wayback Machine (by owner request) return no captures. The archive only has what it crawled; very new or obscure pages may have no captures at all. - This actor reads the archive; it does not submit pages to Save Page Now. ### FAQ **Do I need an Internet Archive account or API key?** No. All endpoints used are public. **How is this different from the Wayback Machine website?** The website shows one URL at a time. Here you can list tens of thousands of captures with filters, pull the content of hundreds of old pages as clean text in one run, and vet a whole list of domains, all exported as JSON, CSV or Excel or sent to your own systems through the Apify API, webhooks and integrations. **Can I get every page of a website that no longer exists?** Run snapshots with `matchType: domain`, `collapse: urlkey`, `statusCodes: ["200"]` to get the URL list, then feed those URLs to the pages mode. **Can AI agents use it?** Yes. It works through the Apify API and the Apify MCP server; the Markdown output is ready for LLM context. **Why did a requested date return a capture from a different day?** You get the capture closest in time to what you asked for. Check `daysFromRequested`; if it matters, list the captures first with the snapshots mode and request exact timestamps. **My run hit HTTP 429 or 503 retries.** The Internet Archive is shared infrastructure and is sometimes overloaded. The actor backs off and retries on its own. For very large jobs, keep `requestIntervalSecs` at 1 or higher and avoid starting many runs at the same moment. ### Changelog - **0.1.7 (October 21, 2026)**: README leads with a dated snapshot run and the four prices; input prefill unchanged. No scraping changes. - **0.1 (October 13, 2026)**: First release. Four modes: snapshot list (CDX with match types, date range, status, MIME and raw filters, collapse, newest first, resume-key paging), archived page content (text, Markdown, raw HTML, meta, links, `every-year`), domain history reports (statistics, sampled homepage titles, redirects, parked signals, URL count) and archive.org item search with file lists. # Actor input Schema ## `mode` (type: `string`): snapshots: list every capture of a URL, path prefix, host or whole domain from the Wayback CDX index, with date, status and MIME filters. pages: the archived page itself (clean text, Markdown and/or raw HTML, title, meta, links) for the dates you choose. domainHistory: one report per domain with first and last seen, captures per year, gaps, homepage title per year, redirects to other sites and parked-domain signals, for SEO and expired-domain research. archiveSearch: search archive.org items (books, films, audio, software, collections). ## `urls` (type: `array`): One per line. Plain domains (example.com), full URLs (https://example.com/blog/), or Wayback links (https://web.archive.org/web/2019.../https://example.com/) for the pages mode. Used by the snapshots, pages and domainHistory modes. ## `matchType` (type: `string`): exact: only this URL. prefix: every URL starting with it (a folder). host: every URL on this host name. domain: every URL on the domain and all its subdomains. Prefix, host and domain matches on big sites return a lot of rows; pair them with a date range, a collapse option or the limit below. ## `collapse` (type: `string`): Keep one capture per hour, day, month or year, one per distinct URL (urlkey, gives the list of archived URLs of a site), or drop captures whose content did not change from the previous one (digest). ## `statusCodes` (type: `array`): Keep only captures with these archived HTTP status codes, e.g. 200, 404, 301 or a class like 3xx. Empty keeps all. ## `mimeTypes` (type: `array`): Keep only these content types, e.g. text/html, application/pdf or image/\* . Empty keeps all. ## `extraFilters` (type: `array`): Raw CDX server filter expressions, passed through as is: field:regex, with ! to negate. Example: !original:.\*.(css|js)$ or length:\[0-9]{5,}. ## `maxSnapshotsPerUrl` (type: `integer`): Stop after this many captures for each input line. Large domains can have millions. ## `newestFirst` (type: `boolean`): Return the most recent captures (up to 10,000) instead of starting from the oldest. ## `dates` (type: `array`): Which version of each page to fetch. Use YYYY, YYYY-MM, YYYY-MM-DD or a 14-digit Wayback timestamp; the capture closest to that moment is returned. Also: latest, first, every-year (one capture per year within the date range). Empty means latest. ## `contentFormats` (type: `array`): text: main content as clean plain text, menus and footers removed. markdown: main content as Markdown with links. html: the raw archived HTML exactly as captured (no Wayback toolbar). ## `includeLinks` (type: `boolean`): Add every link on the archived page (URL, anchor text, internal or external, rel) for backlink and site-structure research. ## `maxContentChars` (type: `integer`): Truncate text, Markdown and HTML to this many characters. 0 keeps everything. ## `dateFrom` (type: `string`): YYYY, YYYY-MM or YYYY-MM-DD. Limits snapshot lists, every-year page samples and domain history years. Empty means from the start of the archive. ## `dateTo` (type: `string`): YYYY, YYYY-MM or YYYY-MM-DD. Empty means up to today. ## `maxTitleSamples` (type: `integer`): How many archived homepages to open per domain (one per year, spread evenly over the domain's life) to record titles, languages, redirects and parked-domain signals. Each sample takes 2 to 4 seconds. 0 gives only the capture statistics (about 1 second per domain). ## `uniqueUrlsUpTo` (type: `integer`): Also count distinct archived URLs on the domain and subdomains, up to this cap (one extra CDX query, slow on very large sites). 0 skips it. ## `searchQuery` (type: `string`): archive.org search syntax, e.g. title:(apollo 11) or creator:(nasa) AND year:\[1960 TO 1972]. ## `mediaType` (type: `string`): Limit the search to one kind of item. ## `collection` (type: `string`): Only items in this archive.org collection identifier, e.g. nasa, prelinger or gutenberg. ## `searchSort` (type: `string`): Result order. ## `maxItems` (type: `integer`): Stop after this many items. ## `includeFileList` (type: `boolean`): Add each item's files with format, size and direct download URL (one extra request per item). ## `requestIntervalSecs` (type: `number`): Minimum pause between requests to the same Internet Archive host. The actor slows down further on its own when archive.org answers 429 or 503. ## `proxyConfiguration` (type: `object`): Not needed normally: the Internet Archive APIs are public. Turn on only if your runs keep getting rate-limited. ## Actor input object example ```json { "mode": "snapshots", "urls": [ "apify.com" ], "matchType": "exact", "collapse": "month", "maxSnapshotsPerUrl": 100, "newestFirst": false, "contentFormats": [ "text", "markdown" ], "includeLinks": false, "maxContentChars": 0, "maxTitleSamples": 20, "uniqueUrlsUpTo": 0, "mediaType": "any", "searchSort": "relevance", "maxItems": 100, "includeFileList": false, "requestIntervalSecs": 1, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `results` (type: `string`): No description ## `snapshotsCsv` (type: `string`): No description ## `pagesCsv` (type: `string`): No description ## `domainHistoryCsv` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "apify.com" ], "collapse": "month", "maxSnapshotsPerUrl": 100 }; // Run the Actor and wait for it to finish const run = await client.actor("varnmill/wayback-machine-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["apify.com"], "collapse": "month", "maxSnapshotsPerUrl": 100, } # Run the Actor and wait for it to finish run = client.actor("varnmill/wayback-machine-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "apify.com" ], "collapse": "month", "maxSnapshotsPerUrl": 100 }' | apify call varnmill/wayback-machine-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,varnmill/wayback-machine-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/KXyZp6u0vjrwBDbPF/builds/TXnvHas0PgbQmdrSg/openapi.json