Internet Archive & Wayback Machine Scraper

Two tools in one actor, both on archive.org's official public APIs, with no API key, no login and no browser:

• Wayback Machine mode lists every archived snapshot of a page or a whole site: capture date, HTTP status, content type, content hash and a ready-to-open snapshot link. Filter by date, status and type, keep only content changes, and optionally download the archived page text.
• Search mode exports archive.org item metadata for books, audio, video, software and web collections, with optional file lists.

Price: $0.50 per 1,000 rows in either mode.

Common uses

• See how a competitor's pricing page, homepage or terms changed over time (collapse = content changes only, with page text).
• Recover the text of a page or site that is now offline.
• Check a domain's history before buying it: first and last capture, gaps, redirects and what it used to host.
• Find every archived URL on a site for SEO migrations, redirect maps or broken link recovery.
• Build reading lists or text corpora from public domain books, or export a collection's catalogue to a spreadsheet.

Quick start: Wayback Machine

{
  "mode": "wayback",
  "urls": ["example.com", "https://www.python.org/"],
  "collapse": "year",
  "from": "2015",
  "to": "2020",
  "statusFilter": "200",
  "maxSnapshotsPerUrl": 50,
  "fetchContent": true
}

Wayback inputs

Name	Default	Description
mode	search	Set to wayback for this mode.
urls		URLs or domains, for example example.com or https://site.com/pricing.
matchType	exact	exact (this URL only), prefix (every URL that starts with it), host (every page on the host), domain (the domain and all subdomains).
from, to	empty	Date range as YYYY, YYYY-MM or YYYY-MM-DD.
collapse	none	day, month or year keep at most one capture per URL per period. digest keeps content changes only: a capture is dropped when its content is identical to the capture before it.
statusFilter	empty	HTTP status, as a regular expression: 200, 3.. for redirects.
mimeFilter	empty	Content type, as a regular expression: text/html, application/pdf.
maxSnapshotsPerUrl	100	Stop after this many snapshots for each URL or domain (up to 100,000).
fetchContent	false	Download each archived page and add page_title, text and text_length.
maxContentChars	20000	Cut long page text to this many characters (0 = no cut).

Wayback output

{
  "type": "wayback_snapshot",
  "input_url": "https://www.python.org/",
  "original_url": "https://www.python.org/",
  "urlkey": "org,python)/",
  "timestamp": "20150101032028",
  "captured_at": "2015-01-01T03:20:28Z",
  "status_code": 200,
  "mimetype": "text/html",
  "digest": "CWG5GMVD...",
  "length": 6411,
  "snapshot_url": "https://web.archive.org/web/20150101032028/https://www.python.org/",
  "raw_snapshot_url": "https://web.archive.org/web/20150101032028id_/https://www.python.org/",
  "page_title": "Welcome to Python.org",
  "text": "Skip to content ...",
  "text_length": 6678,
  "content_error": null
}

page_title, text, text_length and content_error appear only when fetchContent is on. raw_snapshot_url opens the page exactly as archived, without the Wayback toolbar. length is the compressed size in the archive. Exact duplicate index entries are skipped and not charged. A per-target summary is saved to the run's OUTPUT record, and one failing URL does not stop the others.

Search mode

Search mode is the default, so existing inputs keep working unchanged.

{ "query": "mediatype:texts AND subject:python", "sort": "downloads desc", "maxItems": 100 }

Search inputs

Name	Type	Default	Description
query	string	mediatype:texts AND subject:python	Archive.org search query (Lucene syntax). Supports field-scoped queries: creator:tolkien, subject:jazz, collection:librivoxaudio.
mediatype	string	(empty)	Narrow to a media type: texts, audio, movies, software, image, web, data.
sort	string	downloads desc	Sort order. Examples: date desc, titleSorter asc, avg_rating desc.
maxItems	integer	100	Maximum items to return. Hard cap: 10,000.
includeFiles	boolean	false	Also fetch the file manifest for each item. Makes one extra request per item, significantly slower for large result sets.

Query syntax

Archive.org uses Lucene syntax. Some useful patterns:

subject:jazz AND mediatype:audio
creator:"Ernest Hemingway"
collection:gutenberg AND language:english
date:[2020-01-01 TO 2023-12-31]

Search output fields

Field	Type	Description
identifier	string	Unique Archive.org item ID
title	string	Item title
creator	array	Author(s) or creator(s)
description	string	Item description
mediatype	string	texts, audio, movies, software, image, etc.
subject	array	Subject tags
date	string	Original publication date
year	integer	Publication year
language	array	Language(s)
downloads	integer	Total download count
item_size	integer	Total size in bytes
collection	array	Collections the item belongs to
publicdate	string	Date added to Archive.org (ISO 8601)
item_url	string	Direct URL to the item page
files	array	File manifest (only when includeFiles: true)

Search sample output

Result from query mediatype:texts AND subject:python, sorted by downloads desc:

{"identifier":"python_ebooks_2020","title":"Python Ebooks","creator":["strugglez"],"description":"Python Ebooks","mediatype":"texts","subject":["python"],"date":null,"year":null,"language":["Latin"],"downloads":257128,"item_size":49856126105,"collection":["booksbylanguage_latin","booksbylanguage"],"publicdate":"2020-09-27T09:24:26Z","item_url":"https://archive.org/details/python_ebooks_2020","files":null}

Pricing

Pay per result: $0.50 per 1,000 rows, the same in both modes, plus Apify's tiny start fee. 100 snapshots cost $0.05. Downloading page text or file lists does not raise the per-row price, it only makes the run take longer.

Limitations

• Wayback mode lists what the Internet Archive has captured. Pages it never archived, and sites that asked to be excluded, will return no rows.
• collapse: digest compares each capture with the one before it in the index, so it removes repeats in a row, not every duplicate across years.
• Page text is plain text extracted from the archived HTML. It does not run JavaScript, so pages that load their content with scripts may come back mostly empty. Images, PDFs and other non-text files are listed but their content is not downloaded (content_error says not_text).
• With fetchContent on, each snapshot takes about 1.5 seconds, because the actor waits between requests to stay polite to archive.org. Use collapse or a small maxSnapshotsPerUrl for long histories.
• Archive.org caps search mode at 10,000 items per query. Split larger jobs by date or subject.
• Some search fields (date, year, language) are not consistently filled in by contributors, and collection includes personal favourites (prefixed fav-).

Rate limits

Requests run one at a time: 1 second between Wayback index pages, 1.5 seconds between page downloads, 500 ms between search pages and 600 ms between file list requests, with automatic backoff on HTTP 429 and 5xx responses.

Legal

Both modes use archive.org's public endpoints, which need no authentication. Only publicly archived data is returned. Respect the copyright of the archived pages you download.