Internet Archive Metadata Scraper — Bulk archive.org Item Export by Query, Collection & Date

Bulk-export item metadata from the Internet Archive (archive.org) — texts, audio, video, software, images, live concerts and web collections — filtered by full-text/Lucene query, collection, media type, creator, subject and date range, straight into clean tabular rows. Powered by the keyless archive.org Scrape and Advanced Search APIs, it turns the world's largest public digital library into a spreadsheet or dataset in one run. No API key, no login, no cookies, no browser.

---

What does the Internet Archive Metadata Scraper do?

This Actor exports item-level metadata catalogs from archive.org at scale. You give it a raw Lucene query (for example collection:nasa AND mediatype:audio) or a set of structured filters — collection, media type, creator, subject and a date range — and it paginates the archive's public Scrape API by cursor to return up to millions of matching items as flat rows. For each item you get identifier, title, creator, date, publicdate, year, mediatype, collection, downloads, item_size, format, language, subject, description and ready-made details/metadata URLs. When you request a sort (e.g. downloads desc), it automatically switches to the Advanced Search API. Optionally it enriches each row with a per-item file list from the /metadata/ endpoint. Everything runs over a direct, keyless connection — no account, no OAuth, no client secret.

Who is it for?

• Data scientists & ML engineers sourcing large, license-friendly text, audio, image or video corpora for training and evaluation.
• Librarians, archivists & digitization teams cataloguing holdings, auditing collections and reconciling metadata at scale.
• Academic researchers & digital humanists assembling reproducible datasets around a topic, creator, era or collection.
• Media researchers, journalists & documentarians locating public-domain footage, recordings, photos and documents by subject and date.
• Developers & data engineers who need a keyless, scriptable feed of archive.org item metadata for search indexes, catalogs and pipelines.

Use cases

• Build a training corpus — export every texts item in a subject or collection with downloads, language and format to shortlist material for an NLP/LLM dataset.
• Catalog a collection — dump the full item list of a collection (e.g. prelinger, librivoxaudio, internetarcade) into a spreadsheet for inventory and QA.
• Find the most popular items — sort a query by downloads desc to surface the top-viewed films, books or recordings on a topic.
• Time-boxed research slices — pull all items dated 2010-01-01 TO 2015-12-31 for a creator or subject to study a period.
• Monitor new uploads — sort by publicdate desc and schedule the Actor to track what is newly added to a collection or subject over time.

Why use the Internet Archive Metadata Scraper?

• Keyless & login-free — reads archive.org's public Scrape and Advanced Search APIs directly; no API key, OAuth, account or cookies required.
• 19 metadata fields per item — identifier, title, creator, dates, media type, collection, downloads, size, formats, language, subject, description and item URLs, plus optional file lists.
• True bulk export — cursor pagination over the Scrape API returns up to millions of items in a single run (one NASA collection alone holds 200,000+ items).
• Query it your way — one raw Lucene query, or point-and-click filters (collection, media type, creator, subject, date range) combined automatically with AND.
• Sort when you need it — request downloads desc, publicdate desc, date asc and the Actor transparently uses the Advanced Search API.
• Clean, export-ready rows — string-or-array fields are safely normalized to comma-joined values; export to JSON, CSV, Excel, XML or JSONL, or pull via the Apify API and webhooks. Simple pay-per-result pricing.

What data can you extract?

Every matching archive.org item becomes one dataset row. Fields marked (files) are only populated when Fetch per-item file list is enabled.

Field	Type	Description
identifier	string	Unique archive.org item identifier (used in every URL).
title	string	Item title.
creator	string	Creator or author (comma-joined if multiple).
date	string	Item date (publication / recording date).
publicdate	string	Date the item became public on archive.org.
year	string	Year of the item.
mediatype	string	texts, audio, movies, image, software, web, data or collection.
collection	string	Collection(s) the item belongs to (comma-joined).
downloads	number	All-time download / view count.
itemSize	number	Total item size in bytes.
format	string	File formats present in the item, e.g. JPEG, PDF, MP3 (comma-joined).
language	string	Language(s) of the item.
subject	string	Subject / topic tags (comma-joined).
description	string	Item description / abstract.
detailsUrl	string	Public details page: https://archive.org/details/<identifier>.
metadataUrl	string	JSON metadata endpoint: https://archive.org/metadata/<identifier>.
fileCount	number	Number of files in the item. (files)
topFiles	string	Sample of up to 10 file names in the item. (files)

Example row (real archive.org data)

{
  "identifier": "00-042-154",
  "title": "International Space Station exhibit",
  "creator": "NASA/Stennis Space Center",
  "date": "2000-05-26T00:00:00Z",
  "publicdate": "2009-08-26T16:30:57Z",
  "year": "2000",
  "mediatype": "image",
  "collection": "nasa, stennisspacecentercollection",
  "downloads": "1376",
  "itemSize": "1292064",
  "format": "Archive BitTorrent, Item Tile, JPEG, JPEG Thumb, Metadata",
  "language": "",
  "subject": "StenniSphere exhibit, International Space Station, ISS, What -- International Space Station (ISS)",
  "description": "The International Space Station (ISS) exhibit in StenniSphere at John C. Stennis Space Center gives visitors an up-close look at the largest international peacetime project in history.",
  "detailsUrl": "https://archive.org/details/00-042-154",
  "metadataUrl": "https://archive.org/metadata/00-042-154"
}

How to use

You can target the archive in two ways. Both accept the same field selection, sort, maxItems cap and optional file enrichment.

Option A — Structured filters (no query language needed)

1. Create a task for this Actor and open the Input tab.
2. Fill in any combination of Collection, Media Type, Creator, Subject and Date From / Date To. They are combined with AND.
3. (Optional) Set a Sort like downloads desc, choose your Fields, and a Max Items cap.
4. Click Start, then export from the Output tab.

{
  "collection": "nasa",
  "mediaType": "image",
  "maxItems": 200
}

Option B — Raw Lucene query (full control)

1. Put a complete archive.org query in the Query field — this overrides the structured filters.
2. Combine any fields with AND / OR, quote phrases, and use ranges like date:[2015-01-01 TO 2020-12-31].
3. Optionally add a Sort and Max Items, then Start.

{
  "query": "grateful dead AND mediatype:audio AND collection:etree",
  "sort": "downloads desc",
  "maxItems": 500
}

Input parameters

Parameter	Type	Default	Description
query	string	""	Raw Lucene / full-text query. If set, used directly and the filters below are ignored.
collection	string	""	Collection identifier, e.g. nasa, librivoxaudio, prelinger. Adds collection:<value>.
mediaType	select	any	One of any, texts, audio, movies, image, software, web, data, collection.
creator	string	""	Creator / author filter. Adds creator:<value>.
subject	string	""	Subject / topic filter. Adds subject:<value>.
dateFrom	string	""	Start of the item date range (YYYY-MM-DD).
dateTo	string	""	End of the item date range (YYYY-MM-DD). Applied as date:[from TO to].
fields	array	(14 default fields)	Which metadata fields to request. identifier is always included.
sort	string	""	e.g. downloads desc, publicdate desc, date asc. When set, uses Advanced Search (≤10k).
fetchFiles	boolean	false	Also fetch each item's file list from /metadata/ (one extra request per item; slow).
maxItems	integer	1000	Maximum items to export. 0 = unlimited (careful with huge collections).
proxyConfiguration	object	Apify Proxy on	Datacenter proxy by default; a fresh IP is used on retries.

Full input example

{
  "query": "",
  "collection": "librivoxaudio",
  "mediaType": "audio",
  "creator": "",
  "subject": "fiction",
  "dateFrom": "",
  "dateTo": "",
  "fields": ["identifier", "title", "creator", "date", "mediatype", "downloads", "item_size", "format", "language", "subject"],
  "sort": "downloads desc",
  "fetchFiles": false,
  "maxItems": 1000,
  "proxyConfiguration": { "useApifyProxy": true }
}

Output example

A single fully-populated row (with fetchFiles: true, so fileCount and topFiles are present):

{
  "identifier": "cd_greatest-hits_the-grateful-dead",
  "title": "Skeletons From The Closet: The Best Of Grateful Dead",
  "creator": "Grateful Dead",
  "date": "1974-01-01T00:00:00Z",
  "publicdate": "2021-03-14T09:22:41Z",
  "year": "1974",
  "mediatype": "audio",
  "collection": "etree, GratefulDead, stream_only",
  "downloads": "48211",
  "itemSize": "612340288",
  "format": "VBR MP3, Flac, PNG, Metadata, Archive BitTorrent",
  "language": "eng",
  "subject": "Grateful Dead, rock, live, concert",
  "description": "Compilation of Grateful Dead live and studio recordings.",
  "detailsUrl": "https://archive.org/details/cd_greatest-hits_the-grateful-dead",
  "metadataUrl": "https://archive.org/metadata/cd_greatest-hits_the-grateful-dead",
  "fileCount": "34",
  "topFiles": "01 - Uncle John's Band.mp3, 02 - Sugar Magnolia.mp3, cover.png, metadata.xml"
}

Tips for best results

• Start narrow, then scale. Confirm your filters return sensible rows with a small maxItems (e.g. 200), then raise the cap for the full bulk export.
• Use collection: for the highest signal. Collections like nasa, librivoxaudio, prelinger, internetarcade, etree, opensource_movies and gutenberg are large, well-tagged and reliable.
• Combine mediaType with a subject or creator to slice a big collection into the exact media you want.
• Drop the sort for true bulk. Sorting uses Advanced Search, which archive.org caps at ~10,000 results. Leave sort empty to stream unlimited items via the Scrape API.
• Date ranges use the item date field, not the upload date. Use sort: "publicdate desc" if you care about newly added items instead.
• Request only the fields you need. Fewer fields = smaller, faster output — great for very large jobs.
• Enable fetchFiles only when necessary. It adds one request per item and is much slower; keep it off for pure metadata catalogs.
• Quote multi-word values inside a raw query, e.g. subject:"world war ii" or creator:"Mark Twain".
• Escape ranges correctly in raw queries: date:[2000-01-01 TO 2005-12-31] (note the spaces and uppercase TO).
• Deduping is automatic — items are de-duplicated by identifier within a run.

Integrations

Wire the Actor's dataset into your stack with a few clicks:

• Google Sheets — push item catalogs straight into a sheet for review and sharing.
• Slack — get notified when a scheduled run finishes or a monitored collection grows.
• Zapier / Make / n8n — trigger downstream automations from new archive.org items.
• Webhooks — POST results to your own endpoint the moment a run succeeds.
• Schedule — run daily/weekly to keep a fresh mirror of a collection, subject or creator.

Export formats include JSON, CSV, Excel, HTML, XML and JSONL from the Console or the Apify API.

API usage

Run the Actor programmatically with the Apify API or clients. Replace <YOUR_TOKEN> with your Apify API token.

cURL

curl -X POST "https://api.apify.com/v2/acts/logiover~internet-archive-metadata-scraper/runs?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"collection":"nasa","mediaType":"image","maxItems":200}'

Node.js (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_TOKEN>' });

const run = await client.actor('logiover/internet-archive-metadata-scraper').call({
    query: 'collection:librivoxaudio AND mediatype:audio',
    sort: 'downloads desc',
    maxItems: 500,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("<YOUR_TOKEN>")

run = client.actor("logiover/internet-archive-metadata-scraper").call(run_input={
    "collection": "prelinger",
    "mediaType": "movies",
    "maxItems": 1000,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["identifier"], item["title"])

Use with AI agents (MCP)

This Actor works as a tool for AI agents through the Apify MCP (Model Context Protocol) server, so an LLM agent can fetch archive.org item metadata on demand. Point your MCP-capable client (Claude, an agent framework, etc.) at Apify's MCP server and expose logiover/internet-archive-metadata-scraper. Example prompt: "Use the Internet Archive Metadata Scraper to export the 300 most-downloaded audio items in the librivoxaudio collection, sorted by downloads, and give me title, creator and download count."

FAQ

Do I need an API key, an account or a login to scrape the Internet Archive?

No. The Actor reads archive.org's public Scrape and Advanced Search APIs over a direct connection — no API key, OAuth, account, client secret or cookies are required.

How much data can I export in one run?

The Scrape API paginates by cursor and can return millions of items; a single large collection such as nasa holds well over 200,000 items. Use maxItems to control volume and cost, and leave sort empty for unlimited bulk.

Why is there a ~10,000-result limit sometimes?

Only when you request a sort. Sorting uses archive.org's Advanced Search API, which is hard-capped at ~10,000 results. Remove the sort and the Actor switches to the Scrape API, which has no such cap.

What media types can I scrape?

All Internet Archive media types: texts, audio, movies (video), image, software, web, data and collection. Filter to one with Media Type, or leave it any.

How do I write a query for a specific collection and date range?

Either fill in Collection + Date From/To, or use a raw query like collection:prelinger AND date:[1950-01-01 TO 1959-12-31]. Ranges must use uppercase TO and YYYY-MM-DD dates.

My run returned zero items — what happened?

Usually the filters are too specific or a value is misspelled (collection identifiers are exact, e.g. librivoxaudio, not LibriVox). Broaden the query, drop a filter, or verify the collection at archive.org/details/<collection>.

Why are some fields empty or null?

archive.org metadata is contributor-supplied, so not every item has every field (many images lack a language, some items lack a description). Empty fields are omitted from the row rather than guessed.

Can I get the list of files inside each item?

Yes — enable Fetch per-item file list. Each row then includes fileCount and topFiles from the /metadata/ endpoint. It is slower (one extra request per item), so keep it off for pure metadata catalogs.

How do I export the results to CSV or Excel?

Open the Output/Dataset tab and download as CSV, JSON, Excel, XML, HTML or JSONL, or pull the data through the Apify API. Every item is a flat row, so it imports cleanly into spreadsheets and databases.

How is this different from the Wayback Machine URL extractor?

The Wayback tools return captured web-page snapshots (URLs). This Actor returns the digital-library item catalog — the books, audio, film, software and image items hosted on archive.org, with their bibliographic metadata. Different data entirely.

How fresh is the data and how often is it updated?

Results are pulled live from archive.org on every run, so they reflect the current catalog. Schedule the Actor (e.g. sorted by publicdate desc) to keep an up-to-date mirror of a collection or subject.

Is scraping archive.org allowed?

The Actor collects only publicly available metadata via archive.org's own public APIs. You are responsible for using the data in line with the Internet Archive's terms of use and applicable law (see below).

Is it legal?

This Actor extracts only publicly available metadata exposed by the Internet Archive's own public Scrape and Advanced Search APIs, and it does not bypass any authentication or access control. It is intended for legitimate research, cataloguing, analytics and dataset-building. You are responsible for complying with the Internet Archive Terms of Use, the individual items' rights and licenses, applicable copyright, and privacy regulations such as GDPR/CCPA when you store or process the results. Scraping public data is generally lawful, but how you use it is your responsibility.

Related scrapers

Building a broader research or dataset pipeline? Pair this with other logiover Actors:

• Wayback Machine URL Extractor — historical web-page snapshot URLs from the Wayback Machine.
• Reddit Historical Archive Scraper — years of old Reddit posts and comments by date via public archives.
• Hacker News Search Scraper — full-text search over Hacker News stories and comments.
• 👉 Browse all logiover scrapers on Apify Store — 180+ Actors across data, media, social, research and B2B.