Internet Archive Search Scraper

Search Internet Archive and export normalized item metadata, canonical source URLs, and downloadable-file metadata.

Use this Actor to build and refresh archival research corpora without parsing Archive.org pages or downloading large media files during discovery.

It accepts keyword or fielded queries, exact item identifiers, and item URLs.

Each result is one unique Internet Archive item with nested public file metadata.

What does Internet Archive Search Scraper do?

The Actor uses Internet Archive's public structured endpoints to:

• run Internet Archive Advanced Search queries;
• filter searches by media type;
• sort by relevance, downloads, date, or title;
• fetch authoritative metadata for each selected item;
• return canonical item, metadata, download, and thumbnail URLs;
• list bounded public files with format, size, checksum, and direct URL;
• deduplicate items found by more than one input;
• preserve the queries that matched each exported item.

The Actor does not download file bodies.

That keeps discovery runs small and lets your downstream workflow choose which files to retrieve.

Who is this Actor for?

Researchers can create evidence sets for a topic, creator, collection, or media type.

Digital humanities teams can refresh a corpus and compare scheduled dataset exports.

Archivists and preservation teams can inventory public item and file metadata.

Media discovery teams can identify downloadable audio, video, image, text, and software files.

Data engineers can feed normalized records into spreadsheets, databases, object-storage jobs, or ETL pipelines.

Why use it?

Internet Archive has public APIs, but production workflows still need input validation, pagination, item enrichment, retries, deduplication, output schemas, and integration-ready datasets.

This Actor packages those steps into one repeatable run.

Unlike a search-card-only export, each record is enriched through the item metadata endpoint.

Unlike a downloader, it does not transfer every large file automatically.

You receive file-level metadata and can decide what to download later.

What data can I extract?

Field	Meaning
identifier	Stable Internet Archive item identifier
title	Item title when supplied by the archive
creators	Normalized creator list
description	Archive description; may contain source HTML
date, year	Source-provided date values
mediaType	Archive media type such as audio or movies
collections	Collection identifiers, excluding user favorite collections
subjects	Source-provided subject values
languages	Source-provided language values
downloads	Download count when exposed
licenseUrl	Source-provided license link when exposed
sourceUrl	Canonical /details/ item URL
metadataUrl	Public metadata API URL
downloadPageUrl	Canonical item download directory
thumbnailUrl	Internet Archive thumbnail service URL
files	Bounded public file manifest
fileCount	Number of file records included in this result
totalFileSizeBytes	Sum of known sizes for included files
matchedQueries	Input queries that discovered the item
scrapedAt	ISO timestamp for this export

Each files entry includes name, url, format, sizeBytes, md5, sha1, and source.

Missing source metadata is returned as null or an empty array rather than guessed.

How to search Internet Archive

1. Open the Actor input page.
2. Add one or more values to Search queries.
3. Optionally select a media type and result order.
4. Keep Include downloadable files enabled when you need file manifests.
5. Choose a file limit and total item limit.
6. Start the run.
7. Open the default dataset in table, JSON, CSV, Excel, XML, or RSS format.

A simple query such as apollo 11 nasa searches broadly.

An Advanced Search expression such as collection:prelinger targets a collection.

A fielded query such as creator:"NASA" targets archive metadata.

Input parameters

queries

An array of plain or fielded Internet Archive search expressions.

Each query can contain up to 500 characters.

itemUrls

Optional exact archive.org/details/... or archive.org/download/... URLs.

Use this route when you already know the items to enrich.

identifiers

Optional exact identifiers such as Apollo11Audio.

Identifiers and URLs use the same normalization and file limits as search results.

mediaType

Optional search-only filter: texts, movies, audio, software, image, data, web, collection, or etree.

sortBy

Choose relevance, most downloaded, newest, oldest, or title order.

includeFileMetadata

When enabled, return public file metadata nested under each item.

When disabled, item metadata remains available and files is empty.

maxFilesPerItem

Include 0–500 file records per item.

This limit does not download or truncate source files; it only bounds the returned manifest.

maxItems

Return 1–500 unique items across all input routes.

The limit is global, not per query.

Input example

{
  "queries": ["collection:prelinger"],
  "mediaType": "movies",
  "sortBy": "date-desc",
  "includeFileMetadata": true,
  "maxFilesPerItem": 20,
  "maxItems": 5
}

Output example

This abbreviated record comes from an actual run for Apollo11Audio:

{
  "identifier": "Apollo11Audio",
  "title": "Apollo 11",
  "creators": ["NASA"],
  "mediaType": "audio",
  "collections": ["nasaaudiocollection", "nasa", "apolloaudiocollection"],
  "licenseUrl": "http://creativecommons.org/publicdomain/mark/1.0/",
  "sourceUrl": "https://archive.org/details/Apollo11Audio",
  "metadataUrl": "https://archive.org/metadata/Apollo11Audio",
  "downloadPageUrl": "https://archive.org/download/Apollo11Audio",
  "files": [
    {
      "name": "11-03301.mp3",
      "url": "https://archive.org/download/Apollo11Audio/11-03301.mp3",
      "format": "VBR MP3",
      "sizeBytes": 71183839,
      "md5": "b97628e961d17ced7c603cc9cad459b6",
      "source": "original"
    }
  ],
  "fileCount": 3,
  "matchedQueries": []
}

The complete dataset also contains descriptions, subjects, languages, dates, checksums, totals, and timestamps.

How much does it cost to export Internet Archive items?

The Actor uses pay-per-event pricing.

A run currently has a $0.005 start charge plus the plan-specific price for each unique item written to the dataset.

The per-item tier prices are:

Apify plan	Price per unique item
Free	$0.0091908
Bronze	$0.007992
Silver	$0.0062338
Gold	$0.0047952
Platinum	$0.0047952
Diamond	$0.0047952

File entries nested in an item have no separate event charge.

A no-result search incurs the start event but no item events.

At the Bronze rate, 1 item costs about $0.012992, 10 items cost about $0.08492, and 100 items cost about $0.8042, including the start charge.

Apify platform usage is handled according to the pricing displayed for your plan.

Always check the live pricing panel for the exact current tier before a larger run.

Corpus-building workflow

Run a stable fielded query on a schedule.

Export the default dataset to your storage or warehouse.

Use identifier as the durable key.

Compare fields and file checksums between runs.

Queue only newly discovered or changed file URLs for downstream download.

This separates lightweight catalog discovery from potentially expensive media transfer.

Spreadsheet and data-pipeline exports

The default dataset works with Apify's JSON, CSV, Excel, XML, and RSS exports.

Nested arrays such as files are richest in JSON.

For a relational destination, flatten files downstream and keep identifier as the parent key.

Use webhooks or integrations to trigger a pipeline after a successful run.

API usage with cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~internet-archive-search-metadata/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries":["apollo 11 nasa"],"maxItems":10}'

Fetch dataset items after the run finishes using the dataset ID returned by the run API.

API usage with JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/internet-archive-search-metadata').call({
  queries: ['creator:"NASA"'],
  mediaType: 'audio',
  includeFileMetadata: true,
  maxItems: 10,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

API usage with Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("automation-lab/internet-archive-search-metadata").call(
    run_input={"identifiers": ["Apollo11Audio"], "maxFilesPerItem": 10}
)
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Use with Apify MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/internet-archive-search-metadata"

Claude Desktop setup

Add the following remote MCP server in Claude Desktop's MCP configuration.

Cursor setup

Use the same server object in Cursor's MCP settings.

VS Code setup

Add the same remote server URL through your VS Code MCP extension or workspace configuration.

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/internet-archive-search-metadata"
    }
  }
}

Example prompts:

• “Search Internet Archive for five highly downloaded Apollo 11 items.”
• “Export metadata and the first 20 files for movies in collection:prelinger.”
• “Fetch metadata for identifier Apollo11Audio without downloading media.”

Reliability and retry behavior

The Actor uses official public JSON endpoints rather than rendering pages.

Temporary network failures, HTTP 429 responses, and server errors are retried up to three times with bounded backoff.

Invalid input and deterministic client errors are not retried blindly.

Metadata details are fetched in batches of five to limit upstream pressure and memory use.

If an accepted item cannot be enriched, the run fails instead of silently returning an incomplete success.

Limits and source behavior

Archive.org metadata is community-contributed and varies by item.

Some dates, creators, licenses, subjects, sizes, or checksums may be absent.

Descriptions can contain source HTML.

Search ordering and index contents are controlled by Internet Archive and may change.

maxFilesPerItem includes the first public file records in source order; it is not a file-format filter.

Private files are excluded.

The Actor does not bypass access controls or fetch restricted files.

Tips for efficient runs

Start with a small maxItems while refining fielded queries.

Use mediaType to avoid unrelated result types.

Set includeFileMetadata to false when only catalog metadata is needed.

Keep maxFilesPerItem low for items with very large manifests.

Use identifiers or item URLs to refresh a known collection of exact items.

Store the stable identifier rather than relying only on titles.

Responsible use and legality

Internet Archive records can include public-domain, openly licensed, copyrighted, personal, or sensitive material.

A public metadata or download URL does not grant new rights to reuse its contents.

Review each item's license and applicable law before downloading, redistributing, training on, or publishing files.

Respect Internet Archive's terms, service capacity, and removal decisions.

Do not use this Actor to circumvent authentication, privacy controls, or access restrictions.

You are responsible for your input, processing purpose, retention policy, and downstream use.

Troubleshooting

Why did my query return no items?

Test the expression in Internet Archive Advanced Search, remove overly narrow fields, and confirm the selected media type matches the catalog records.

A genuine empty search succeeds with zero item events.

Why did the run reject an item URL?

Use an archive.org/details/<identifier> or archive.org/download/<identifier> URL.

Wayback Machine snapshot URLs are a different product and are not accepted.

Why are some fields null or empty?

The source item did not expose those fields.

The Actor does not invent missing metadata.

Why are fewer files present than on the item page?

Increase maxFilesPerItem up to 500.

Private files and directory placeholders are excluded.

Does this Actor download the files?

No.

It returns direct public URLs and metadata so a separate authorized workflow can select downloads.

FAQ

Can I use Advanced Search syntax?

Yes.

Queries are passed to Internet Archive Advanced Search, and the optional media-type filter is applied consistently.

Can I mix queries, URLs, and identifiers?

Yes.

The Actor deduplicates exact identifiers and applies one global maxItems limit.

How do I monitor changes?

Schedule the same stable input, export each dataset, and compare records by identifier and file checksum.

The Actor produces snapshots; it does not send alerts itself.

Is every Internet Archive file downloadable?

No.

The Actor lists public file metadata exposed by the source and excludes records marked private.

Availability and rights remain controlled by Internet Archive and the item owner.

Related Automation Lab Actors

For scholarly literature rather than mixed archival media, use arXiv Paper Search & Export.

For downloadable public webpage source rather than archive item metadata, use Public Webpage HTML Downloader.

For official LiDAR catalog metadata, use Public LiDAR Dataset Catalog Exporter.

These Actors solve different source-specific jobs and do not replace Internet Archive catalog search.