# Internet Archive Search Scraper (`automation-lab/internet-archive-search-metadata`) Actor

Search Internet Archive and export normalized item metadata, canonical source URLs, and bounded downloadable-file manifests for archival research and corpus building.

- **URL**: https://apify.com/automation-lab/internet-archive-search-metadata.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** Education
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.80 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Internet Archive Search Scraper

Search Internet Archive and export normalized item metadata, canonical source URLs, and downloadable-file metadata.

Use this Actor to build and refresh archival research corpora without parsing Archive.org pages or downloading large media files during discovery.

It accepts keyword or fielded queries, exact item identifiers, and item URLs.

Each result is one unique Internet Archive item with nested public file metadata.

### What does Internet Archive Search Scraper do?

The Actor uses Internet Archive's public structured endpoints to:

- run Internet Archive Advanced Search queries;
- filter searches by media type;
- sort by relevance, downloads, date, or title;
- fetch authoritative metadata for each selected item;
- return canonical item, metadata, download, and thumbnail URLs;
- list bounded public files with format, size, checksum, and direct URL;
- deduplicate items found by more than one input;
- preserve the queries that matched each exported item.

The Actor does not download file bodies.

That keeps discovery runs small and lets your downstream workflow choose which files to retrieve.

### Who is this Actor for?

**Researchers** can create evidence sets for a topic, creator, collection, or media type.

**Digital humanities teams** can refresh a corpus and compare scheduled dataset exports.

**Archivists and preservation teams** can inventory public item and file metadata.

**Media discovery teams** can identify downloadable audio, video, image, text, and software files.

**Data engineers** can feed normalized records into spreadsheets, databases, object-storage jobs, or ETL pipelines.

### Why use it?

Internet Archive has public APIs, but production workflows still need input validation, pagination, item enrichment, retries, deduplication, output schemas, and integration-ready datasets.

This Actor packages those steps into one repeatable run.

Unlike a search-card-only export, each record is enriched through the item metadata endpoint.

Unlike a downloader, it does not transfer every large file automatically.

You receive file-level metadata and can decide what to download later.

### What data can I extract?

| Field | Meaning |
| --- | --- |
| `identifier` | Stable Internet Archive item identifier |
| `title` | Item title when supplied by the archive |
| `creators` | Normalized creator list |
| `description` | Archive description; may contain source HTML |
| `date`, `year` | Source-provided date values |
| `mediaType` | Archive media type such as `audio` or `movies` |
| `collections` | Collection identifiers, excluding user favorite collections |
| `subjects` | Source-provided subject values |
| `languages` | Source-provided language values |
| `downloads` | Download count when exposed |
| `licenseUrl` | Source-provided license link when exposed |
| `sourceUrl` | Canonical `/details/` item URL |
| `metadataUrl` | Public metadata API URL |
| `downloadPageUrl` | Canonical item download directory |
| `thumbnailUrl` | Internet Archive thumbnail service URL |
| `files` | Bounded public file manifest |
| `fileCount` | Number of file records included in this result |
| `totalFileSizeBytes` | Sum of known sizes for included files |
| `matchedQueries` | Input queries that discovered the item |
| `scrapedAt` | ISO timestamp for this export |

Each `files` entry includes `name`, `url`, `format`, `sizeBytes`, `md5`, `sha1`, and `source`.

Missing source metadata is returned as `null` or an empty array rather than guessed.

### How to search Internet Archive

1. Open the Actor input page.
2. Add one or more values to **Search queries**.
3. Optionally select a media type and result order.
4. Keep **Include downloadable files** enabled when you need file manifests.
5. Choose a file limit and total item limit.
6. Start the run.
7. Open the default dataset in table, JSON, CSV, Excel, XML, or RSS format.

A simple query such as `apollo 11 nasa` searches broadly.

An Advanced Search expression such as `collection:prelinger` targets a collection.

A fielded query such as `creator:"NASA"` targets archive metadata.

### Input parameters

#### `queries`

An array of plain or fielded Internet Archive search expressions.

Each query can contain up to 500 characters.

#### `itemUrls`

Optional exact `archive.org/details/...` or `archive.org/download/...` URLs.

Use this route when you already know the items to enrich.

#### `identifiers`

Optional exact identifiers such as `Apollo11Audio`.

Identifiers and URLs use the same normalization and file limits as search results.

#### `mediaType`

Optional search-only filter: texts, movies, audio, software, image, data, web, collection, or etree.

#### `sortBy`

Choose relevance, most downloaded, newest, oldest, or title order.

#### `includeFileMetadata`

When enabled, return public file metadata nested under each item.

When disabled, item metadata remains available and `files` is empty.

#### `maxFilesPerItem`

Include 0–500 file records per item.

This limit does not download or truncate source files; it only bounds the returned manifest.

#### `maxItems`

Return 1–500 unique items across all input routes.

The limit is global, not per query.

### Input example

```json
{
  "queries": ["collection:prelinger"],
  "mediaType": "movies",
  "sortBy": "date-desc",
  "includeFileMetadata": true,
  "maxFilesPerItem": 20,
  "maxItems": 5
}
```

### Output example

This abbreviated record comes from an actual run for `Apollo11Audio`:

```json
{
  "identifier": "Apollo11Audio",
  "title": "Apollo 11",
  "creators": ["NASA"],
  "mediaType": "audio",
  "collections": ["nasaaudiocollection", "nasa", "apolloaudiocollection"],
  "licenseUrl": "http://creativecommons.org/publicdomain/mark/1.0/",
  "sourceUrl": "https://archive.org/details/Apollo11Audio",
  "metadataUrl": "https://archive.org/metadata/Apollo11Audio",
  "downloadPageUrl": "https://archive.org/download/Apollo11Audio",
  "files": [
    {
      "name": "11-03301.mp3",
      "url": "https://archive.org/download/Apollo11Audio/11-03301.mp3",
      "format": "VBR MP3",
      "sizeBytes": 71183839,
      "md5": "b97628e961d17ced7c603cc9cad459b6",
      "source": "original"
    }
  ],
  "fileCount": 3,
  "matchedQueries": []
}
```

The complete dataset also contains descriptions, subjects, languages, dates, checksums, totals, and timestamps.

### How much does it cost to export Internet Archive items?

The Actor uses pay-per-event pricing.

A run currently has a **$0.005 start charge** plus the plan-specific price for each unique item written to the dataset.

The per-item tier prices are:

| Apify plan | Price per unique item |
| --- | ---: |
| Free | $0.0091908 |
| Bronze | $0.007992 |
| Silver | $0.0062338 |
| Gold | $0.0047952 |
| Platinum | $0.0047952 |
| Diamond | $0.0047952 |

File entries nested in an item have no separate event charge.

A no-result search incurs the start event but no item events.

At the Bronze rate, 1 item costs about **$0.012992**, 10 items cost about **$0.08492**, and 100 items cost about **$0.8042**, including the start charge.

Apify platform usage is handled according to the pricing displayed for your plan.

Always check the live pricing panel for the exact current tier before a larger run.

### Corpus-building workflow

Run a stable fielded query on a schedule.

Export the default dataset to your storage or warehouse.

Use `identifier` as the durable key.

Compare fields and file checksums between runs.

Queue only newly discovered or changed file URLs for downstream download.

This separates lightweight catalog discovery from potentially expensive media transfer.

### Spreadsheet and data-pipeline exports

The default dataset works with Apify's JSON, CSV, Excel, XML, and RSS exports.

Nested arrays such as `files` are richest in JSON.

For a relational destination, flatten files downstream and keep `identifier` as the parent key.

Use webhooks or integrations to trigger a pipeline after a successful run.

### API usage with cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~internet-archive-search-metadata/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries":["apollo 11 nasa"],"maxItems":10}'
```

Fetch dataset items after the run finishes using the dataset ID returned by the run API.

### API usage with JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/internet-archive-search-metadata').call({
  queries: ['creator:"NASA"'],
  mediaType: 'audio',
  includeFileMetadata: true,
  maxItems: 10,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

### API usage with Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("automation-lab/internet-archive-search-metadata").call(
    run_input={"identifiers": ["Apollo11Audio"], "maxFilesPerItem": 10}
)
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)
```

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/internet-archive-search-metadata"
```

#### Claude Desktop setup

Add the following remote MCP server in Claude Desktop's MCP configuration.

#### Cursor setup

Use the same server object in Cursor's MCP settings.

#### VS Code setup

Add the same remote server URL through your VS Code MCP extension or workspace configuration.

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/internet-archive-search-metadata"
    }
  }
}
```

Example prompts:

- “Search Internet Archive for five highly downloaded Apollo 11 items.”
- “Export metadata and the first 20 files for movies in collection:prelinger.”
- “Fetch metadata for identifier Apollo11Audio without downloading media.”

### Reliability and retry behavior

The Actor uses official public JSON endpoints rather than rendering pages.

Temporary network failures, HTTP 429 responses, and server errors are retried up to three times with bounded backoff.

Invalid input and deterministic client errors are not retried blindly.

Metadata details are fetched in batches of five to limit upstream pressure and memory use.

If an accepted item cannot be enriched, the run fails instead of silently returning an incomplete success.

### Limits and source behavior

Archive.org metadata is community-contributed and varies by item.

Some dates, creators, licenses, subjects, sizes, or checksums may be absent.

Descriptions can contain source HTML.

Search ordering and index contents are controlled by Internet Archive and may change.

`maxFilesPerItem` includes the first public file records in source order; it is not a file-format filter.

Private files are excluded.

The Actor does not bypass access controls or fetch restricted files.

### Tips for efficient runs

Start with a small `maxItems` while refining fielded queries.

Use `mediaType` to avoid unrelated result types.

Set `includeFileMetadata` to `false` when only catalog metadata is needed.

Keep `maxFilesPerItem` low for items with very large manifests.

Use identifiers or item URLs to refresh a known collection of exact items.

Store the stable identifier rather than relying only on titles.

### Responsible use and legality

Internet Archive records can include public-domain, openly licensed, copyrighted, personal, or sensitive material.

A public metadata or download URL does not grant new rights to reuse its contents.

Review each item's license and applicable law before downloading, redistributing, training on, or publishing files.

Respect Internet Archive's terms, service capacity, and removal decisions.

Do not use this Actor to circumvent authentication, privacy controls, or access restrictions.

You are responsible for your input, processing purpose, retention policy, and downstream use.

### Troubleshooting

#### Why did my query return no items?

Test the expression in Internet Archive Advanced Search, remove overly narrow fields, and confirm the selected media type matches the catalog records.

A genuine empty search succeeds with zero item events.

#### Why did the run reject an item URL?

Use an `archive.org/details/<identifier>` or `archive.org/download/<identifier>` URL.

Wayback Machine snapshot URLs are a different product and are not accepted.

#### Why are some fields null or empty?

The source item did not expose those fields.

The Actor does not invent missing metadata.

#### Why are fewer files present than on the item page?

Increase `maxFilesPerItem` up to 500.

Private files and directory placeholders are excluded.

#### Does this Actor download the files?

No.

It returns direct public URLs and metadata so a separate authorized workflow can select downloads.

### FAQ

#### Can I use Advanced Search syntax?

Yes.

Queries are passed to Internet Archive Advanced Search, and the optional media-type filter is applied consistently.

#### Can I mix queries, URLs, and identifiers?

Yes.

The Actor deduplicates exact identifiers and applies one global `maxItems` limit.

#### How do I monitor changes?

Schedule the same stable input, export each dataset, and compare records by `identifier` and file checksum.

The Actor produces snapshots; it does not send alerts itself.

#### Is every Internet Archive file downloadable?

No.

The Actor lists public file metadata exposed by the source and excludes records marked private.

Availability and rights remain controlled by Internet Archive and the item owner.

### Related Automation Lab Actors

For scholarly literature rather than mixed archival media, use [arXiv Paper Search & Export](https://apify.com/automation-lab/arxiv-paper-search-export).

For downloadable public webpage source rather than archive item metadata, use [Public Webpage HTML Downloader](https://apify.com/automation-lab/public-webpage-html-downloader).

For official LiDAR catalog metadata, use [Public LiDAR Dataset Catalog Exporter](https://apify.com/automation-lab/public-lidar-dataset-catalog-exporter).

These Actors solve different source-specific jobs and do not replace Internet Archive catalog search.

# Actor input Schema

## `queries` (type: `array`):

Internet Archive Advanced Search queries. Plain keywords and fielded queries such as creator:"NASA" are supported.

## `itemUrls` (type: `array`):

Optional archive.org/details/... or archive.org/download/... item URLs to export directly.

## `identifiers` (type: `array`):

Optional exact Internet Archive identifiers, such as Apollo11Audio.

## `mediaType` (type: `string`):

Apply an Internet Archive media-type filter to every search query.

## `sortBy` (type: `string`):

Choose relevance, popularity, publication date, or title order.

## `includeFileMetadata` (type: `boolean`):

Include file names, formats, sizes, checksums, and direct download URLs from each item's public metadata.

## `maxFilesPerItem` (type: `integer`):

Maximum public file records included in each item. Set 0 to omit files while retaining item metadata.

## `maxItems` (type: `integer`):

Maximum unique Internet Archive items across all queries, URLs, and identifiers.

## Actor input object example

```json
{
  "queries": [
    "apollo 11 nasa"
  ],
  "mediaType": "",
  "sortBy": "relevance",
  "includeFileMetadata": true,
  "maxFilesPerItem": 20,
  "maxItems": 10
}
```

# Actor output Schema

## `overview` (type: `string`):

Open the normalized item records in the overview table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "apollo 11 nasa"
    ],
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/internet-archive-search-metadata").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["apollo 11 nasa"],
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/internet-archive-search-metadata").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "apollo 11 nasa"
  ],
  "maxItems": 10
}' |
apify call automation-lab/internet-archive-search-metadata --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/internet-archive-search-metadata"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5HvXbzPKQeLEYSndg/builds/jwElEarewL31g5aYc/openapi.json