Internet Archive and Wayback Machine scraper

Search the Internet Archive's 40M+ items, pull the full record and file list for any item, and look up historical snapshots of any URL in the Wayback Machine. One actor, four jobs, all running on archive.org's official public APIs. No API key, no login, nothing that rate-limits into breakage.

Four modes

Pick one in the What to do field:

• Search items: query the catalog of books, audio, video, software, images, web archives, and data. Filter by media type, sort by relevance, downloads, date, or title.
• Item metadata + files: give it archive.org identifiers and get the complete metadata plus every file, with format, size, and download base.
• Wayback snapshots (all): list every capture of a URL from the Wayback Machine, with timestamp, status, MIME type, and the direct snapshot link. Filter by date range.
• Wayback closest snapshot: find the single capture nearest a date you name, or the most recent one.

Search example

{
  "mode": "search",
  "searchQueries": ["machine learning", "public domain films"],
  "mediaType": "texts",
  "sortBy": "downloads desc",
  "maxResults": 50
}

Each item comes back with identifier, title, creator, description, media type, collection, date, year, language, subject tags, download count, item size, file count, formats, and ready-made links to the details page, the download page, and the thumbnail.

Wayback example

{
  "mode": "waybackCaptures",
  "urls": ["example.com", "nasa.gov"],
  "waybackFrom": "2010",
  "waybackTo": "2015",
  "maxResults": 100
}

You get one row per snapshot: the capture timestamp, an ISO date, the original URL, the HTTP status, the content type, and a waybackUrl that opens the archived page.

Who uses it

Researchers and librarians pulling public-domain sources, journalists and OSINT analysts checking what a page said on a given date, digital preservation teams cataloguing archived sites and old software, and anyone building a dataset from public archive metadata without clicking through archive.org by hand.

How to find an item identifier

Open any item on archive.org. The identifier is the last part of the URL: for archive.org/details/nasa_apollo11, the identifier is nasa_apollo11. Put that in the Item identifiers field for metadata mode.

Notes

Everything here reads public data through documented archive.org endpoints: the advanced search API, the item metadata API, the Wayback CDX API, and the availability API. It stores no credentials and pulls nothing private. Broad search terms return huge result sets, so narrow with a media type or a more specific query when you can.