Internet Archive Scraper — Wayback Machine + Library Search + Item Metadata

The fastest way to turn archive.org into structured data. Paste a URL or domain for its full Wayback Machine snapshot history, a keyword to search the Internet Archive library (books, movies, software, music), or an item ID for complete metadata + file lists — every query is auto-classified, no mode switches, no configuration.

Why Use This Scraper?

• ✅ Four data sources in one actor — Wayback snapshots, library search, item metadata, closest-snapshot lookups
• ✅ Paste anything: URLs, bare domains, keywords, item IDs — each line auto-routed to the right API
• ✅ Date-range filters for both snapshot history and library search
• ✅ Flat JSON rows ready for CSV / spreadsheet export — no nested-array digging
• ✅ No proxies needed and none charged — archive.org's APIs are open, so runs are fast and cheap
• ✅ Duplicate-snapshot collapsing for clean domain-history timelines

Overview

The Internet Archive Scraper is built for SEO specialists, researchers, digital archivists, and developers who need structured data from archive.org without stitching together four different APIs.

The output is a stream of flat rows, and the row shape follows the query type: a domain query yields snapshot rows, a keyword yields search-result rows, an item ID yields one item row, and a snapshotOf: lookup yields one availability row. All rows carry a type discriminator and echo the originating query, so mixed runs stay easy to split downstream.

Supported Inputs

Query types (auto-detected per line)

Query type	Pattern	Example	Rows produced
Domain history	bare domain	example.com	one row per Wayback snapshot of the whole host
URL history	full URL	https://example.com/page	one row per snapshot of that exact URL
Path prefix	trailing /*	example.com/blog/*	snapshots of everything under the path
Wayback URL	web.archive.org/web/...	https://web.archive.org/web/2020/https://example.com	snapshot history of the embedded URL
Library search	free text	nasa apollo mission	one row per matching archive.org item
Item metadata	item URL or item: prefix	https://archive.org/details/InformationM or item:InformationM	one metadata row with the full file list
Closest snapshot	snapshotOf:URL[@YYYYMMDD]	snapshotOf:example.com@20200601	one availability row with the nearest archived copy

Copy-pasteable input

{
  "queries": [
    "example.com",
    "nasa apollo mission",
    "item:InformationM",
    "snapshotOf:example.com@20200601"
  ],
  "maxItems": 200
}

Unsupported inputs

• ❌ Content that was excluded from the Wayback Machine by the site owner (robots-excluded domains return no snapshots)
• ❌ Borrow-only / lending-library book content (metadata is returned, but not restricted file downloads)
• ❌ Full-text passages from inside books or videos — rows describe items and files, not their inner content

Use Cases

Audience	Use case
SEO teams	Audit a domain's history before buying it; recover content from expired domains; track competitor site changes over the years
Researchers	Bulk-export library holdings on a topic — books, films, recordings — with dates, collections, and download counts
Legal / compliance teams	Document what a page said on a specific date with a citable snapshot URL
Digital archivists	Inventory item collections with full metadata and file-level detail (formats, sizes, checksums)
Developers	Feed snapshot timelines or library metadata into apps without writing four API clients

How It Works

1. Input — add any mix of URLs, domains, keywords, item IDs, or snapshotOf: lookups, one per line
2. Classify — each line is auto-routed: URL/domain → Wayback CDX index, free text → library search, item ID → metadata API, snapshotOf: → availability API
3. Fetch — the actor calls archive.org's open APIs over plain HTTP with polite concurrency, retries, and back-off
4. Filter — date ranges, mediatype, sort order, and duplicate-collapsing are applied server-side where the API supports it
5. Output — flat, typed JSON rows land in the dataset, exportable as JSON, CSV, Excel, or XML

Input Configuration

Input fields

Field	Type	Required	Notes
queries	array<string>	yes	One query per line; each auto-classified (see Supported Inputs)
from	string (date)	optional	Range start; snapshot timestamps in CDX mode, publicdate in search mode. Defaults to 1996-01-01
to	string (date)	optional	Range end; empty = up to now. Also the target date for snapshotOf: without @date
mediatype	string enum	optional	Search mode only: texts, movies, audio, image, software, data, web, collection, etree
collapseSnapshots	boolean	optional	CDX mode only: collapse consecutive identical-content snapshots. Default false
searchSort	string enum	optional	Search mode only: downloads desc/asc, publicdate desc/asc, date desc/asc, titleSorter asc. Empty = relevance
maxItems	integer	optional	Global cap on rows across all queries. Default 100
maxConcurrency	integer	optional	Parallel queries, default 2, hard cap 5 — archive.org is a nonprofit, be kind

Common scenarios

1. Full snapshot history of a domain, deduplicated

{
  "queries": ["example.com"],
  "collapseSnapshots": true,
  "maxItems": 1000
}

2. Most-downloaded NASA images added since 2020

{
  "queries": ["nasa"],
  "mediatype": "image",
  "from": "2020-01-01",
  "searchSort": "downloads desc",
  "maxItems": 500
}

3. What did a page look like on a given date?

{
  "queries": ["snapshotOf:cnn.com@20080915"]
}

Output Overview

Every row is flat (no nested objects except small arrays like collections) and carries:

• type — snapshot, searchResult, item, or availability
• query — the input line that produced the row, for easy grouping
• Snapshot rows — timestamp, ISO date, original URL, ready-to-open snapshotUrl, MIME type, HTTP status, content digest, size
• Search-result rows — identifier, item URL, title, mediatype, creator, dates, downloads, size, collections, subjects, plus totalFound for the whole query
• Item rows — full metadata (title, description, publisher, license, collections) plus file count and per-file download URLs, formats, sizes, and MD5 checksums
• Availability rows — whether an archived copy exists near the requested date, and the closest snapshot's URL, timestamp, and status

Output Samples

Domain query → snapshot row

{
  "type": "snapshot",
  "query": "example.com",
  "urlkey": "com,example)/",
  "timestamp": "20020120142510",
  "snapshotDate": "2002-01-20T14:25:10Z",
  "originalUrl": "http://example.com:80/",
  "snapshotUrl": "https://web.archive.org/web/20020120142510/http://example.com:80/",
  "mimeType": "text/html",
  "statusCode": 200,
  "digest": "HT2DYGA5UKZCPBSFVCV3JOBXGW2G5UUA",
  "contentLength": 1792,
  "scrapedAt": "2026-07-17T10:01:03.171Z"
}

Keyword query → search-result row

{
  "type": "searchResult",
  "query": "nasa",
  "identifier": "prgd14higzbtwbvzrdeclelzpz0xjxbaagvlkcaj",
  "itemUrl": "https://archive.org/details/prgd14higzbtwbvzrdeclelzpz0xjxbaagvlkcaj",
  "title": "Small Steps, Giant Leaps: Episode 25, NASA Technology Transfer",
  "mediatype": "audio",
  "creator": "Small Steps, Giant Leaps",
  "date": "2019-12-10T00:00:00Z",
  "publicDate": "2023-01-16T20:23:02Z",
  "downloads": 1,
  "itemSizeBytes": 18627727,
  "collections": ["podcasts_mirror_bobarchives", "podcasts_mirror"],
  "subjects": ["Podcast", "APPEL", "NASA", "projectmanagement"],
  "totalFound": 78788,
  "scrapedAt": "2026-07-17T08:37:45.951Z"
}

Item query → metadata row (trimmed)

{
  "type": "item",
  "query": "item:InformationM",
  "identifier": "InformationM",
  "itemUrl": "https://archive.org/details/InformationM",
  "title": "Information Machine, The",
  "mediatype": "movies",
  "description": "Applies graphic sensitivity to medium in cartoon form, and traces the history of storing and analyzing information…",
  "publisher": "Eames (Charles And Ray)",
  "date": "1958",
  "licenseUrl": "http://creativecommons.org/licenses/publicdomain/",
  "collections": ["prelinger"],
  "itemSizeBytes": 684174158
  /* + filesCount and files[] with per-file name, format, size, md5, downloadUrl */
}

snapshotOf: query → availability row

{
  "type": "availability",
  "query": "snapshotOf:example.com@20200601",
  "url": "example.com",
  "requestedTimestamp": "20200601",
  "available": true,
  "snapshotUrl": "https://web.archive.org/web/20200601235352/https://example.com/",
  "snapshotTimestamp": "20200601235352",
  "snapshotStatus": "200"
}

Key Output Fields

Common

• type, query, scrapedAt

Snapshot (CDX)

• timestamp, snapshotDate, originalUrl, snapshotUrl
• mimeType, statusCode, digest, contentLength

Search result

• identifier, itemUrl, title, mediatype, creator, description
• date, year, publicDate, addedDate
• downloads, itemSizeBytes, collections[], subjects[], totalFound

Item metadata

• identifier, title, description, publisher, licenseUrl, language
• filesCount, files[] — per-file name, format, size, md5, downloadUrl

Availability

• available, requestedTimestamp, snapshotUrl, snapshotTimestamp, snapshotStatus

FAQ

Do I need proxies?

No. The actor uses archive.org's open, unauthenticated APIs — there is no anti-bot wall, so nothing about proxies appears in the input and no proxy costs are added to your runs.

How does the actor know which mode to use?

Each query line is classified by shape: anything URL- or domain-like goes to the Wayback CDX index, item: prefixes and archive.org item URLs go to the metadata API, snapshotOf: goes to the availability API, and everything else is a library search. You can mix all four in one run.

Why did my snapshotOf: lookup return available: false for a page I know is archived?

Archive.org's availability endpoint is occasionally flaky — the identical query can return empty once and succeed minutes later, both with HTTP 200. Re-run the query, or use the domain/URL history mode (CDX) instead, which is reliable and returns every snapshot rather than just the closest one.

How many results can a library search return?

Standard pagination is capped by archive.org at around 10,000 results per query. For bigger harvests, split the query with from/to date ranges. totalFound on each row tells you the full match count.

What does maxItems limit exactly?

It is a global budget across all queries in the run, not per query. If one domain's snapshot history fills the cap, later queries in the list are skipped — the run log shows how many queries were processed.

Can I download the actual files or page content?

Rows include ready-to-use URLs (snapshotUrl for archived pages, per-file downloadUrl for item files), but the actor deliberately does not download file contents — it delivers the structured index so you fetch only what you need.

Can I scrape borrow-only books or private collections?

No. The actor reads public metadata and public snapshot indexes only. Restricted-access content (lending-library books, darked items) is out of scope.

Why do some snapshot rows have statusCode: null?

Those are warc/revisit records — the Wayback crawler noted the content was unchanged rather than storing a new copy. The digest still identifies the content version.

Support

• For issues or feature requests, please use the Issues tab on the actor's Apify Console page.
• Author's website: https://muhamed-didovic.github.io/
• Email: muhamed.didovic@gmail.com

Additional Services

• Need deep pagination beyond 10k results, extra CDX filters (status code, MIME type), or scheduled domain-history monitoring? Drop a line — custom builds are quick on this codebase.
• Custom output shapes, private builds, or direct API access (no Apify fee, just a usage fee): muhamed.didovic@gmail.com

Explore More Scrapers

If you found this useful, you might also like:

• Trustpilot Scraper — company reviews at scale for reputation research
• Glassdoor Scraper — employer reviews, salaries, and interview data
• VRBO Scraper — vacation-rental properties with calendar, rates, and full review history

Full list at apify.com/memo23.

---

⚠️ Disclaimer

This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by the Internet Archive (a 501(c)(3) non-profit) or any of its projects, including the Wayback Machine. All trademarks mentioned are the property of their respective owners.

The scraper accesses only publicly available data via archive.org's open APIs — no authenticated endpoints, no restricted-access collections, and no content behind the archive.org login. Users are responsible for ensuring their use complies with archive.org's Terms of Use, the licenses attached to individual items, applicable data-protection law (GDPR, CCPA, etc.), and any contractual obligations of their own organization. Please keep concurrency low — the Internet Archive is a nonprofit library.

---

SEO Keywords

internet archive scraper, archive.org scraper, wayback machine scraper, scrape wayback machine, wayback machine API, archive.org API, CDX API scraper, website history scraper, domain history checker, snapshot history export, expired domain research, old website content recovery, archive.org search export, internet archive metadata, digital library data, web archive data extraction, SEO domain audit, historical website data, Apify internet archive, wayback snapshots CSV