Internet Archive Scraper API — Wayback Machine snapshot monitor

Search Internet Archive (archive.org) items and list Wayback Machine snapshots through the Internet Archive's own official public APIs. You get one flat JSON row per item or capture. Turn on "Only new since last run" and the Actor becomes a scheduled Internet Archive monitor: each run returns only items added to a collection or search since the previous run, or only new Wayback captures of a URL or domain.

Unofficial: not affiliated with, endorsed by, or operated by the Internet Archive. This Actor returns metadata and links only, through the documented APIs: advancedsearch, Item Metadata API, and the Wayback CDX server. It never downloads item files, full text, or archived page content, and it doesn't log in or bypass anything.

Your use of Internet Archive data is subject to the Internet Archive's Terms of Use; you are responsible for how you use results.

What this Internet Archive scraper does

• archive.org search scraper and API. Filter Internet Archive items by keyword (full Lucene query syntax supported), collection, media type (texts, movies, audio, software, image, data, web…), creator, subject, item date range, and date added to archive.org. Sort by newest added, newest public, item date, or most downloaded.
• Internet Archive monitor (delta mode). Opt-in onlyNewSinceLastRun returns only items you haven't received yet, so a daily schedule on a collection or saved search costs you only for genuinely new uploads.
• Wayback Machine snapshots. mode: "waybackSnapshots" lists Wayback Machine captures of an exact URL, a URL prefix, a host, or a whole domain, with a from/to date range, HTTP status and MIME filters, and "only content changes" (collapse by digest). Delta mode works here too, so you can be alerted when the Wayback Machine has new snapshots of a site.
• Flat, clean output that's easy to export to CSV, Excel, Google Sheets, or a database. An optional raw pass-through is available.
• Optional full metadata. fetchFullMetadata adds the Item Metadata API record for each item (uploader/operator fields are dropped).
• Privacy built in. Email addresses are redacted ([email redacted]) in every output field of both modes: titles, creators, subjects, descriptions, raw records, and full metadata. Users' favorites lists (fav-<username>) are never returned. In Wayback URLs, emails, URL credentials, and the values of secret-looking query parameters (token, key, password, auth, session, sig, …) are replaced.
• AI-agent ready. Works as a tool in Apify's MCP server, LangChain, CrewAI and the OpenAI Agents SDK. Opt-in outputFormat: "compact" returns small, LLM-friendly rows (see Use with AI agents / MCP).
• Polite by design. Requests run one at a time with a descriptive User-Agent, and the Actor backs off and retries on HTTP 429/5xx and maintenance pages. It uses no proxies, cookies, or logins.

Who it's for

• Researchers, librarians, and digital humanities teams tracking new uploads to Internet Archive collections
• Journalists and OSINT/brand-monitoring analysts who want to know when the Wayback Machine captured a page or a site, and when its content changed
• SEO and web teams auditing a domain's archive history (status codes, MIME types, content changes over time)
• Data engineers who need a stable archive.org API feed in JSON/CSV without writing their own client

Quick start

1. Click Try for free and run the prefilled input (keyword nasa, media type texts, 25 items).
2. To monitor a collection, set collections (for example ["nasa"]), turn on onlyNewSinceLastRun, and add an Apify Schedule (daily works well).
3. For Wayback Machine snapshots, set mode to waybackSnapshots and url to example.com, and optionally turn on collapseDigest.

An empty input {} also works. It uses the same default scope as the prefill (keyword nasa, media type texts, 25 items). The unfiltered stream of newest archive.org uploads is opt-in via includeAllRecentUploads.

Input

Search mode (default, mode: "search")

Field	Type	Notes
query	string	Keywords or IA/Lucene syntax, e.g. nasa, "climate change", title:(apollo) AND year:[1960 TO 1975]
collections	string[]	Collection identifiers, OR'ed (e.g. nasa, prelinger, librivoxaudio)
mediatype	enum	texts, movies, audio, software, image, data, web, collection, etree
creator / subject	string	Exact-phrase match
dateFrom / dateTo	string	The item's own date: YYYY, YYYY-MM or YYYY-MM-DD
addedFrom / addedTo	string	When the item was added to archive.org (addeddate)
sortBy	enum	addeddate_desc (default), publicdate_desc, date_desc, downloads_desc
descriptionMaxLength	int	Description snippet length, HTML stripped (default 300; 0 = omit)
fetchFullMetadata	bool	Adds fullMetadata, filesCount, itemSize via the Item Metadata API (one extra request per item)
includeAllRecentUploads	bool	Opt-in, default off. Only when query and all filters are empty: the newest additions across all of archive.org (last ~7 days), including unmoderated community uploads. When off, empty input uses the default scope (nasa / texts)

Wayback Machine mode (mode: "waybackSnapshots")

Field	Type	Notes
url	string	Required. A page URL or a domain, e.g. example.com or https://example.com/pricing (domain is accepted as an alias)
matchType	enum	exact (default), prefix, host, domain (host plus subdomains)
from / to	string	Capture date range: YYYY, YYYYMM, YYYYMMDD, YYYY-MM-DD or a 14-digit timestamp
collapseDigest	bool	Only content changes: keeps the first capture of each run of identical content
statusFilter	string	e.g. 200, 2xx, !404
mimeTypeFilter	string	e.g. text/html, application/pdf, !image/jpeg

Shared

Field	Default	Notes
maxItems	25	Max dataset rows per run (hard cap 5,000). You pay per row.
onlyNewSinceLastRun	false	Delta mode (see below)
lookbackHours	24	Delta mode: re-checks this window for late-indexed items (never re-charged)
includeRaw	false	Attaches the original search document or CDX row under raw
outputFormat	full	compact = key fields only with texts ≤ 200 chars, for AI agents (search: identifier, title, creator, mediatype, collections, date, addedDate, itemUrl, description; Wayback: capturedAt, originalUrl, statusCode, mimeType, digest, snapshotUrl, urlRedacted). includeRaw and fetchFullMetadata are ignored in compact mode. Same price per row.

Input examples

{ "query": "nasa", "mediatype": "texts", "maxItems": 25 }

{ "collections": ["prelinger"], "mediatype": "movies", "dateFrom": "1950", "dateTo": "1959", "sortBy": "downloads_desc", "maxItems": 100 }

{ "collections": ["nasa"], "onlyNewSinceLastRun": true, "maxItems": 200 }

{ "mode": "waybackSnapshots", "url": "example.com", "collapseDigest": true, "statusFilter": "200", "maxItems": 50 }

{ "mode": "waybackSnapshots", "url": "example.com", "matchType": "domain", "from": "2026-10-01", "mimeTypeFilter": "text/html", "maxItems": 500 }

Output example

Search mode (one row per Internet Archive item):

{
  "identifier": "HealthYo1953",
  "title": "Health: Your Posture",
  "creator": "Centron Corporation",
  "mediatype": "movies",
  "collections": ["prelinger"],
  "subjects": ["Health and hygiene"],
  "date": "1953-01-01",
  "year": "1953",
  "addedDate": "2002-07-16T00:00:00Z",
  "publicDate": "2002-07-16T00:00:00Z",
  "description": "Young girl's mirror image teaches her fundamentals of good posture.",
  "language": null,
  "downloads": 10636630,
  "itemSize": 803125822,
  "itemUrl": "https://archive.org/details/HealthYo1953",
  "metadataUrl": "https://archive.org/metadata/HealthYo1953",
  "thumbnailUrl": "https://archive.org/services/img/HealthYo1953",
  "source": "archive.org",
  "mode": "search"
}

Wayback Machine mode (one row per snapshot/capture):

{
  "timestamp": "20261008110753",
  "capturedAt": "2026-10-08T11:07:53Z",
  "originalUrl": "https://example.com/",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "U2352G3H6BFNEDPBNQI5XGOBSQAN4W4J",
  "length": 1089,
  "snapshotUrl": "https://web.archive.org/web/20261008110753/https://example.com/",
  "rawSnapshotUrl": "https://web.archive.org/web/20261008110753id_/https://example.com/",
  "urlkey": "com,example)/",
  "urlRedacted": false,
  "source": "web.archive.org",
  "mode": "waybackSnapshots"
}

Each run also writes a summary to the default key-value store under OUTPUT, containing ok, mode, count, requestUrl, apiTotal (search), truncated, any auto-applied window, and the delta-mode fields below.

Only new since last run (delta mode)

Built for scheduled / recurring runs, so you aren't re-charged for rows you already received.

1. The first run with the flag on is a baseline. It behaves like a normal run (newest first, capped by maxItems) and then saves a watermark: the newest addedDate (search) or capture timestamp (Wayback) it pushed, plus the IDs already pushed at that moment.
2. Later runs query from the watermark onward, oldest new item first, and push only items you haven't received yet. If there's nothing new, the run still succeeds with an empty dataset (OUTPUT.message = "No new items since last run").
3. Lookback for late indexing. Internet Archive search and the Wayback CDX index can list something a little after its addeddate or capture time. Each delta run therefore re-checks lookbackHours (default 24) before the watermark and emits only IDs it hasn't returned before. Set it to 0 for a strict watermark.
4. The watermark is updated only after rows were pushed successfully, so a failed run doesn't move it.

Where the state lives: a named key-value store internet-archive-monitor-state in the Apify account that runs the Actor (yours). There's one record per filter combination (WATERMARK-<hash of your filters>), so two schedules with different filters, or different Wayback URLs/domains, never share state. Changing any filter starts a new baseline. To reset, delete that record (Console → Storage → Key-value stores).

OUTPUT fields in delta mode: onlyNewSinceLastRun, watermarkKey, baselineRun, watermarkBefore, watermarkAfter, queriedFrom, lookbackHours, skippedAlreadySeen, truncated (more new rows are waiting and arrive on the next run), and on a baseline run olderMatchesSkipped.

Delta-mode caveats (please read):

• "New" means newly added or newly captured. Metadata edits to items you already received aren't re-emitted.
• If more new rows exist than maxItems, the oldest new ones come first and the rest arrive on the next run (OUTPUT.truncated = true).
• Empty-filter schedules can miss items. With includeAllRecentUploads on and no filters, search mode follows the whole archive.org firehose (roughly 1,500+ uploads per hour as of Oct 2026) inside a rolling ~7-day window. If maxItems is below the volume per run, or a schedule is paused for a long time, the backlog grows, and anything that falls out of the rolling window is never backfilled. Use real filters (a collection, query, or media type), or raise maxItems, if you see truncated: true run after run.
• A baseline doesn't backfill history older than its newest maxItems rows (olderMatchesSkipped: true tells you more existed).
• Wayback prefix/host/domain delta runs scan every capture since the watermark (up to 20,000 rows per run). Above that, the run stops with a clear error instead of silently skipping captures. Narrow the URL, add filters, or schedule more often. A baseline with no from uses the last ~7 days.
• With collapseDigest on, the Actor remembers the last content digest per URL (up to 5,000 URLs), so an unchanged page isn't reported again on the next run. A late-indexed capture older than the last capture already processed for that URL is not emitted.
• State is per Apify account and per filter set. Two runs with identical filters started at the same moment can both return the same rows.

Wayback Machine mode details

• exact returns the newest captures first. prefix/host/domain return rows in CDX order (by URL, then by time) for normal runs. Delta runs re-sort by capture time.
• collapseDigest gives one row per content change: the first capture of each run of identical content, per URL. For exact, this is computed client-side from a block of the newest raw captures, so the timestamps show when a change was first captured.
• snapshotUrl opens the archived page in the Wayback Machine. rawSnapshotUrl (id_) is the original archived response without the Wayback toolbar.
• URL redaction. If a captured URL contains an email address, user:password@ credentials, or a secret-looking query parameter (names containing token, key, password, secret, auth, session, sig/signature, credential, …), those parts are replaced (REDACTED / email-redacted, credentials removed) in originalUrl, urlkey, snapshotUrl, rawSnapshotUrl, and raw, and the row gets urlRedacted: true. The rest of the URL is unchanged, and the original unredacted URL is never output. A link that only had credentials removed still opens the capture. A link with a redacted query parameter or path segment (for example /files/email-redacted.pdf) may show Wayback's "not archived" page, because Wayback looks up the exact URL. Check urlRedacted: true on those rows. Captures that differ only in a redacted value share one urlkey, so they're treated as the same URL (de-duplicated and tracked as one URL for change detection). Image filenames such as logo@2x.png aren't emails and are left intact.
• Very popular URLs (with hundreds of thousands of captures) can take 1–2 minutes per query because the CDX server reads the whole index for that URL. The Actor waits and retries patiently.

Use with AI agents / MCP

This Actor works as a ready-made tool for AI agents. An agent passes a search (or a URL in Wayback mode) and gets back one JSON row per archive.org item or Wayback capture: metadata and links only.

Apify MCP server. Add the Actor as a tool by listing it in the tools parameter of the hosted server (https://mcp.apify.com):

https://mcp.apify.com?tools=brightish_ascent/internet-archive-monitor

MCP client config (Claude Desktop, Cursor, VS Code and other MCP clients). You sign in to Apify with OAuth on first connect:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=brightish_ascent/internet-archive-monitor"
    }
  }
}

• The tool is named brightish_ascent--internet-archive-monitor. Its arguments are the input fields above.
• A call returns the run status and a datasetId. The agent then reads the rows with the get-dataset-items tool (limit, offset, fields).
• With the default tool set, an agent can also run it via call-actor with actor: "brightish_ascent/internet-archive-monitor".
• Local stdio server: npx @apify/actors-mcp-server --tools brightish_ascent/internet-archive-monitor, with APIFY_TOKEN set.

Agent frameworks.

• LangChain: ApifyActorsTool("brightish_ascent/internet-archive-monitor") from langchain-apify.
• CrewAI: ApifyActorsTool(actor_name="brightish_ascent/internet-archive-monitor") from crewai_tools.
• OpenAI Agents SDK: MCPServerStreamableHttp with the MCP URL above and an Authorization: Bearer <APIFY_TOKEN> header.
• LlamaIndex: ApifyActor(token).load_data(actor_id="brightish_ascent/internet-archive-monitor", run_input={...}, dataset_mapping_function=...).

The LangChain and CrewAI tools return every dataset row inline, so keep maxItems small.

Example agent prompts:

Find archive.org books and documents about the Apollo program from 1960 to 1975. Use the Internet Archive tool with query: "apollo", mediatype: "texts", dateFrom: "1960", dateTo: "1975", sortBy: "downloads_desc", maxItems: 10, outputFormat: "compact", and list title, date and itemUrl.

When did the Wayback Machine capture example.com, and when did the page change? Use the Internet Archive tool with mode: "waybackSnapshots", url: "example.com", collapseDigest: true, statusFilter: "200", maxItems: 10, outputFormat: "compact".

Cost per call. You pay $0.00005 per run plus $0.0015 per returned row:

maxItems	Max cost per call
10	≈ $0.015
25 (default)	≈ $0.038
100	≈ $0.15

Zero matches cost only the start fee. With call-actor, you can also cap spend with maxTotalChargeUsd in callOptions.

Tips for agents:

• Use specific filters (query, collections, mediatype, creator, dates) and a small maxItems (5–20). Empty input uses the default scope (nasa / texts).
• Use outputFormat: "compact" to keep rows short. Leave includeRaw and fetchFullMetadata off.
• In Wayback mode, matchType: "exact" (the default) returns the newest captures first. Add collapseDigest: true to get one row per content change.
• For "check again later" tasks, set onlyNewSinceLastRun: true so repeat runs return only new items or captures.
• Results are metadata and links only. Cite itemUrl or snapshotUrl. Your use of Internet Archive data is subject to the Internet Archive's Terms of Use.

Pricing

Pay per event (PAY_PER_EVENT):

Event	Price (USD)	Notes
Actor start (apify-actor-start)	$0.00005	Per Actor start, charged per GB of run memory: 1 event for runs up to 1 GB (the default is 256 MB), plus 1 per extra GB
Result row (apify-default-dataset-item)	$0.0015	Per item or snapshot pushed: $1.50 per 1,000 results

Examples (default 256 MB memory): 25 items ≈ $0.00005 + 25 × $0.0015 ≈ $0.04. 1,000 snapshots ≈ $1.50. A delta run that finds nothing new costs only the start event.

FAQ

Is this the official Internet Archive API? It's a client for the Internet Archive's official public APIs (advancedsearch, Item Metadata, Wayback CDX). It isn't made by, affiliated with, or endorsed by the Internet Archive.

Does it download books, videos, or files? No. It returns metadata and links (itemUrl, metadataUrl, snapshotUrl). Whether you may use an item's content depends on that item's own rights and license. Check the item page.

How do I monitor a collection for new uploads? Set collections, turn on onlyNewSinceLastRun, and create an Apify Schedule (daily or hourly). Connect a webhook, Slack, email, or Google Sheets integration to get alerts.

How do I get alerted when a website gets a new Wayback Machine snapshot? Use mode: "waybackSnapshots", url: "yoursite.com", onlyNewSinceLastRun: true, and optionally collapseDigest: true to be alerted only when the archived content changed.

Why did a run return 0 rows? In delta mode, nothing new arrived since the last run. That's a successful, start-fee-only run. Otherwise, your filters matched nothing. Check OUTPUT.requestUrl.

Why did a run fail with "Internet Archive rejected the search query"? Your query has a Lucene syntax error (for example, an unclosed parenthesis). The exact upstream message is shown.

Can I use the full Internet Archive query syntax? Yes. query is passed through, so title:(…), AND/OR/NOT, ranges, and wildcards work. The other filters are added with AND.

What does an empty input do? It runs the same default scope as the prefill: keyword nasa, media type texts, 25 items. To get the unfiltered newest uploads across all of archive.org (community uploads included), set includeAllRecentUploads: true.

Does it return personal data? It returns public bibliographic metadata (titles, creators, subjects, descriptions). Email addresses, including common obfuscated forms such as name [at] domain [dot] com, name AT domain DOT com and name at gmail.com, are redacted in every field; users' favorites lists are excluded, and uploader/operator fields are dropped from full metadata. Creator names stay, because they're the bibliographic record.

Will it ever download files or full text? No. This Actor returns metadata and links only and won't add file-download or full-text features.

Who governs my use of the data? Your use of Internet Archive data is subject to the Internet Archive's Terms of Use; you are responsible for how you use results.

Limitations (honest list)

• Data quality is whatever uploaders entered: titles, dates, and creators can be missing or inconsistent. Item date is free-form upstream and normalized to YYYY-MM-DD where possible.
• Search covers metadata only, not full text inside books.
• One run scans at most 10,000 search rows and returns at most 5,000.
• The Wayback CDX API is sometimes slow or briefly unavailable (HTTP 503/504, "Temporarily Offline"). The Actor retries with backoff, but a long outage can still fail a run.
• truncated in normal (non-delta) runs means more matches exist beyond maxItems. For exact Wayback queries this is an estimate.
• The Internet Archive's collections contain material of every kind, including content some people find objectionable. Apart from the default empty-input scope, results aren't filtered for content.
• Email redaction is pattern-based. It catches standard and URL-encoded addresses (including ones followed by a file extension, like jane@example.com.pdf); bracketed, spaced and full-width forms (name [at] domain [dot] com, name (at) domain.com, name @ domain.com, name＠domain．com); name at gmail.com-style text for well-known email providers (Gmail, Yahoo, Outlook/Hotmail, iCloud, Proton, GMX, Yandex, QQ, 163, Naver, web.de, free.fr, mail.ru/inbox.ru, …), including name -at- gmail.com, name _at_ gmail.com and name - at - gmail dot com; hyphenated forms (name-at-domain-dot-com); and word forms (name AT domain DOT com). Remaining gaps: name at domain.com (bare "at") with a non-provider domain and no "dot" word is not redacted, so that prose such as "available at archive.org" stays intact (a spaced name -at- domain.com or name _at_ domain.com is redacted for any domain); some rare word forms are kept to protect prose (an ending that is also a common English word with a single label, like name at example dot me, and ALL-CAPS plain-word text like NAME AT EXAMPLE DOT COM); and unusual obfuscations may still slip through. An occasional non-email string that looks like an address may be redacted. Text such as "sign in at gmail.com" may be redacted even when it isn't an address: we deliberately prefer over-redaction to leaking an address. Provider names that are also shops, portals or ISPs (bol, sky, uol, terra, frontier, charter, telstra, kakao, mts, optimum, duck, aim, email.com, xtra, excite, prodigy) are only redacted in name at … text when the name part looks like an address (it contains a digit, ., _, + or %) or the at-token is unambiguous ([at], (at), @, or a spaced -at- / _at_), so prose like "buy it at bol.com" stays intact. A plain-word name with a bare "at" on those domains (joao at uol.com.br, also in ALL CAPS or with a hyphenated name like mary-ann at bol.com.br) is a known gap. Addresses written with @ are always redacted. Social handles such as youtube.com/@name and image names such as logo@2x.png are kept. Fediverse handles (@user@mastodon.social) look like email addresses and are redacted.

License

MIT (Actor code). Internet Archive data and items remain subject to the Internet Archive's terms and each item's own rights.