Internet Archive & Wayback Machine Scraper

What does Internet Archive Scraper do?

Search the Internet Archive library and the Wayback Machine in one Actor — for OSINT, journalism, and research. Pull books, movies, audio, software, images, collections, plus item metadata, file lists, capture history, and closest-snapshot lookups. Rows land as JSON (export CSV / Excel / HTML / XML from the dataset). No login, no Archive API key.

Unofficial. This Actor is not affiliated with, endorsed by, or maintained by the Internet Archive. It calls their public search, metadata, CDX, and availability APIs.

Why this Internet Archive scraper?

• 🗞️ OSINT, journalism, and research — historical pages, robots.txt, collections, institutional media
• 📚 Library search across texts, movies, audio, software, images, web, data
• 📦 Item metadata + optional per-file rows (name, format, size in bytes, download URL)
• 🕰️ Wayback CDX snapshots (timestamp, status, MIME, digest, snapshot URL)
• 🎯 Closest archived copy (snapshotOf:url@YYYYMMDD)
• 🔎 Lucene filters: collection, creator, language, date range, sort
• 🧾 Cursor scrape API for deep result sets (not a single page)
• 💸 $0.49 per 1,000 results — below other Store Archive/Wayback Actors

How to use it

1. Open this Actor and click Start (prefilled Sketch Show S1E3 item).
2. Replace queries with keywords, collection:…, item IDs, or URLs.
3. Leave Mode = Auto unless you need to force Search / Metadata / Files / Snapshots.
4. Export the dataset, or open the Issues tab on this Store page if a run fails.

{
  "queries": ["https://archive.org/details/the-sketch-show-s1-e3"],
  "maxItems": 5,
  "mode": "auto"
}

Query cheat sheet (auto mode)

You paste	What you get
the sketch show or collection:nasa	Library item rows
item:the-sketch-show-s1-e3 or https://archive.org/details/the-sketch-show-s1-e3	Full metadata for that identifier
bbc.co.uk or https://www.bbc.co.uk/	Wayback snapshot rows
bbc.co.uk/news/*	Prefix match on that path
snapshotOf:bbc.co.uk@20150601	Closest capture near that date

How much does it cost?

Pay per event (PPE): a tiny Actor start fee plus each result row. On Free, results are $0.49 / 1,000. Paid Apify plans are lower (see the Pricing tab). Platform usage is included. Default run is 5 rows.

Input

Field	Description
queries	One query per line (required unless you only set filters)
mode	auto (default), search, metadata, files, snapshots, available
mediaType	texts, movies, audio, image, software, data, web, …
collection / creator / language	Extra Lucene filters
dateFrom / dateTo	Item date (search) or capture time (Wayback)
maxItems	Cap across all queries (prefill 5)
includeFiles	Extra dataset rows for each file on an item
collapseSnapshots	Collapse consecutive identical Wayback captures

Sample output (library item)

{
  "kind": "item",
  "identifier": "the-sketch-show-s1-e3",
  "title": "The Sketch Show - Series 1 - Episode 3 (2001)",
  "mediaType": "movies",
  "year": "2001",
  "itemSizeBytes": 1327829379,
  "archiveUrl": "https://archive.org/details/the-sketch-show-s1-e3",
  "thumbnailUrl": "https://archive.org/services/img/the-sketch-show-s1-e3",
  "downloadUrl": "https://archive.org/download/the-sketch-show-s1-e3",
  "source": "archive.org"
}

Sample output (Wayback snapshot)

{
  "kind": "snapshot",
  "originalUrl": "http://www.bbc.co.uk/",
  "timestamp": "19970412193203",
  "statusCode": "200",
  "mimeType": "text/html",
  "snapshotUrl": "https://web.archive.org/web/19970412193203/http://www.bbc.co.uk/",
  "source": "web.archive.org"
}

FAQ

Do you download books, videos, or WARC files?

No. Rows are catalog metadata, file listings, and snapshot index lines. Download URLs are included so you can fetch bytes yourself. Lending-library / borrow-only files stay restricted on archive.org.

Is this the live web, or history?

Library search is the live archive.org catalog. Wayback modes are historical captures, not a live crawl of the original site.

Do I need proxies?

Usually no. The Archive’s APIs are public. Enable Apify Proxy only if your IP is throttled.

Is this for OSINT or journalism?

Yes — public catalog search and Wayback snapshot history (timestamps, status, digest, replay URLs). No login and no personal-data enrichment. A cheap demo is the Google robots.txt CDX example on this Actor’s Examples tab.

Can AI agents call this?

Yes — Actor ID bakos_bence/internet-archive via Apify API or MCP.

Related Actors

Actor	Use together when…
Internet Archive Scraper	(this Actor) — library + Wayback

Profile: bakos_bence on Apify.

For AI agents & LLM apps

Paste this Actor’s Apify Store URL into an agent that can use Apify (MCP, Claude, Cursor, etc.). With an Apify account and API token, the agent can set input, run the Actor, and fetch the dataset.

Actor ID: bakos_bence/internet-archive

Purpose: OSINT, journalism, and research on archive.org and the Wayback Machine — library search, item metadata, file lists, CDX snapshots, closest-copy lookup.

Minimal input:

{ "queries": ["https://archive.org/details/the-sketch-show-s1-e3"], "maxItems": 5 }

SEO keywords

internet archive scraper, archive.org api, wayback machine scraper, wayback cdx, internet archive search, archive.org metadata, snapshot history, digital library scraper, wayback osint, data journalism archive.org, apify internet archive