# Internet Archive Scraper (`cordate_jebel/internet-archive-scraper`) Actor

Search archive.org via its public API and export flat rows.

- **URL**: https://apify.com/cordate\_jebel/internet-archive-scraper.md
- **Developed by:** [Andres Clap](https://apify.com/cordate_jebel) (community)
- **Categories:** Other, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Internet Archive Scraper

Search **[archive.org](https://archive.org)** — books, films, audio, software,
images, web captures — **or** list every **Wayback Machine** snapshot of a URL.
Both run on archive.org's **public APIs with no API key** and return flat,
spreadsheet-ready rows.

### What does Internet Archive Scraper do?

Pick a `mode`:

- **`search`** — run an archive.org advanced-search `query` (Lucene syntax) over
  the whole archive and get one row per item: identifier, title, creator, media
  type, download counts, URLs, and (optionally) full metadata.
- **`wayback`** — give it a `waybackUrl` and get every archived snapshot of that
  URL from the [Wayback Machine CDX index](https://web.archive.org): timestamp,
  a direct snapshot URL, MIME type, HTTP status, content digest and size —
  perfect for tracking how a page changed over years.

### Why use it?

- **Research & archiving** — pull large, filtered slices of the world's biggest
  free library as JSON/CSV/Excel.
- **Change tracking / OSINT** — see the full history of any web page: pricing
  pages, ToS, product listings, news articles, deleted content.
- **No key, no login** — archive.org publishes these endpoints for exactly this.
- Runs on the Apify platform: schedule it, hit it via API, pipe into
  integrations, monitor runs.

### How to use it

1. Choose **Mode** — *Search archive.org* or *Wayback Machine snapshots of a URL*.
2. Fill the fields for that mode (a query, or a URL).
3. Set **Max items**.
4. **Start** — results land in the dataset; download as JSON, CSV, Excel or XML.

### Input

#### mode = `search`

| field | notes |
|---|---|
| `query` | advanced-search query, Lucene syntax. e.g. `subject:"machine learning" AND mediatype:texts` |
| `mediatype` | optional filter: `texts`, `movies`, `audio`, `image`, `software`, `web`, … |
| `sort` | `downloads desc` (default), this week / month, newest, recently added, title, rating |
| `startPage` | first result page (100 / page) |
| `fetchMetadata` | also call `/metadata/<id>` per item for description, subjects, license, file / review counts (slower) |

#### mode = `wayback`

| field | notes |
|---|---|
| `waybackUrl` | the URL to look up, e.g. `https://apify.com/pricing` |
| `waybackMatchType` | `exact` (default), `prefix` (path and below), `host`, `domain` (+ subdomains) |
| `waybackFrom` / `waybackTo` | date bounds — `YYYY`, `YYYYMM` or `YYYYMMDD` |
| `waybackCollapse` | skip consecutive snapshots with an unchanged content digest (default on) |
| `waybackOnlyOk` | keep only HTTP 200 snapshots (default on) |

Shared: `maxItems` (default 100).

#### Examples

```json
{ "mode": "search", "query": "collection:nasa", "mediatype": "image", "maxItems": 500 }
```

```json
{ "mode": "wayback", "waybackUrl": "https://apify.com/", "waybackMatchType": "prefix",
  "waybackFrom": "2016", "waybackTo": "2020", "maxItems": 1000 }
```

### Output

**Search rows:** `identifier`, `title`, `creator`, `date`, `publicdate`,
`mediatype`, `collection` (`fav-*` stripped), `language`, `subjects`,
`downloads`, `downloads_last_week`, `downloads_last_month`, `num_reviews`,
`avg_rating`, `item_size_bytes`, `details_url`, `download_url`, `thumbnail_url`
— plus with `fetchMetadata`: `description`, `licenseurl`, `uploader`,
`files_count`, `num_files`, `review_count`.

**Wayback rows:** `query_url`, `timestamp`, `archived_at` (ISO), `snapshot_url`,
`original_url`, `url_key`, `mimetype`, `status_code`, `digest`, `length_bytes`.

A run summary is written to the key-value store as `OUTPUT`.

### Pricing

The Actor only makes lightweight JSON requests — a typical run is a few hundred
kB and well within the Apify free tier. Wayback CDX responses for a busy site
can be large; use `waybackCollapse`, the date bounds, and `maxItems` to keep
runs small.

### Tips

- archive.org caps search `page * rows` at ~10 000. For bigger pulls, split the
  query by date range / collection / media type and run slices.
- CDX `matchType=domain` on a large site can return hundreds of thousands of
  rows — always pair it with a date range.
- Query syntax: <https://archive.org/advancedsearch.php> ·
  CDX docs: <https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server>

### FAQ & disclaimers

archive.org's search and CDX endpoints are public and intended for programmatic
use; this Actor adds a descriptive User-Agent and paces its requests. It stores
no personal data. Found a bug or want a field added? Open an issue on the
Actor's **Issues** tab.

# Actor input Schema

## `mode` (type: `string`):

'search' = query the whole archive (books, film, audio, software, images). 'wayback' = list every archived snapshot of one URL from the Wayback Machine.

## `query` (type: `string`):

archive.org advanced-search query, Lucene syntax. Examples: subject:"machine learning"  |  collection:nasa AND mediatype:image  |  creator:"NASA" AND date:\[2000-01-01 TO 2010-12-31]

## `mediatype` (type: `string`):

Optional convenience filter ANDed into the query.

## `sort` (type: `string`):

How archive.org should order results before pagination.

## `startPage` (type: `integer`):

First result page to fetch (100 items per page). Use to resume or skip ahead.

## `fetchMetadata` (type: `boolean`):

For every result, also call /metadata/<id> to add description, subjects, license, file count and review count. Slower (one extra request per item).

## `waybackUrl` (type: `string`):

The URL whose archived snapshots you want, e.g. https://apify.com or example.com/pricing

## `waybackMatchType` (type: `string`):

exact = that URL only. prefix = URL and everything under its path. host = the whole hostname. domain = the domain and all sub-domains.

## `waybackFrom` (type: `string`):

Earliest snapshot, as YYYY, YYYYMM or YYYYMMDD.

## `waybackTo` (type: `string`):

Latest snapshot, as YYYY, YYYYMM or YYYYMMDD.

## `waybackCollapse` (type: `boolean`):

Skip consecutive snapshots whose content digest is unchanged.

## `waybackOnlyOk` (type: `boolean`):

Drop redirects, 404s and errors.

## `maxItems` (type: `integer`):

Stop after this many rows. archive.org caps search deep-pagination near 10000.

## Actor input object example

```json
{
  "mode": "search",
  "query": "subject:\"machine learning\" AND mediatype:texts",
  "mediatype": "",
  "sort": "downloads desc",
  "startPage": 1,
  "fetchMetadata": false,
  "waybackUrl": "https://apify.com/",
  "waybackMatchType": "exact",
  "waybackCollapse": true,
  "waybackOnlyOk": true,
  "maxItems": 100
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "subject:\"machine learning\" AND mediatype:texts",
    "waybackUrl": "https://apify.com/"
};

// Run the Actor and wait for it to finish
const run = await client.actor("cordate_jebel/internet-archive-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "subject:\"machine learning\" AND mediatype:texts",
    "waybackUrl": "https://apify.com/",
}

# Run the Actor and wait for it to finish
run = client.actor("cordate_jebel/internet-archive-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "subject:\\"machine learning\\" AND mediatype:texts",
  "waybackUrl": "https://apify.com/"
}' |
apify call cordate_jebel/internet-archive-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cordate_jebel/internet-archive-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/etKOsXWvyZbccCtEd/builds/frCAqgcyuUvJhcqMR/openapi.json