Internet Archive Scraper

Every item on archive.org was catalogued by whoever uploaded it, which is why a radio broadcast can carry an exact date and no creator at all. This runs a query against archive.org's own advancedsearch endpoint and writes what comes back, one row per item: identifier, title, creator, year, media type, download count, subject tags, description and a direct item URL. You can pin the search to one media type and pick the sort order.

That endpoint is public, so there's no login and no key to get.

What you get

{
  "identifier": "apolloaudiocollection",
  "title": "Apollo",
  "creator": null,
  "year": null,
  "date": null,
  "mediaType": "collection",
  "downloads": 1259223,
  "subjects": [],
  "description": null,
  "publicdate": "2010-12-06T19:02:28Z",
  "url": "https://archive.org/details/apolloaudiocollection"
}

That's a real row, nulls and all. Uploader-supplied metadata is patchy, and that is the thing to know before you build anything on top of this.

identifier is the archive's primary key. If you're going to download the files afterwards, that's the field you need: https://archive.org/download/<identifier> lists everything in the item.

Input

{
  "query": "nasa apollo",
  "mediaType": "movies",
  "sort": "downloads",
  "maxItems": 500
}

The query field takes the Lucene syntax archive.org itself uses:

{
  "query": "title:(grateful dead) AND year:[1977 TO 1980]",
  "mediaType": "audio",
  "sort": "date",
  "maxItems": 1000
}

• query — required. Plain keywords or Lucene operators.
• mediaType — texts, audio, movies, software, image, web, data, collection, or empty for any.
• sort — downloads (default, most-downloaded first), date (newest item date), publicdate (most recently added to archive.org), or relevance.
• maxItems — default 100, ceiling 10,000. The actor pages 100 per request until it has that many or the result set runs out.

Optionally write each item into Notion with notionConnector.

Limits

Metadata is uploader-supplied, so fields are often null. creator, year, date, description and subjects depend entirely on whoever uploaded the item bothering to fill them in. Collections and older uploads are the worst for this. Don't build a pipeline that assumes year exists.

mediaType: "collection" returns collections, not their contents. A collection is a container. If you search without a media type filter you'll get collections mixed in with items, as in the row above. Filter by media type if you want actual books or recordings.

This searches and returns metadata. It does not download files. Use identifier against archive.org's own download paths for that.

10,000 items per run. Deep pagination is also slow: archive.org's search endpoint gets sluggish once you're thousands of rows into a result set.

Proxy

Off by default, which saves you proxy credits. The advancedsearch API is public, needs no auth and has no anti-bot, so there's nothing for a proxy to solve. Enable Apify Proxy only if you're running very high volume and hitting IP rate limits.

Billing

$2.00 per 1,000 items, and there is no run-start fee.

You pay per item row delivered. A query that matches nothing costs nothing.