Internet Archive Scraper — $0.0005/Item · Books, Texts & Media

Search the Internet Archive (archive.org) by keyword or Lucene query and get structured metadata back for books, texts, audio, video, software, and images: title, creator, description, download count, file size, subjects, and the direct download link, no scraping library or API key required on your end.

Why use this scraper

The Internet Archive has a public search API, but it returns raw, deeply nested JSON built for its own website, with inconsistent fields across media types and no simple way to cap results or filter by media type without reading its query-syntax documentation first. This Actor:

• Wraps that search in a single flat JSON schema that is the same whether you searched for a book, a movie, or a piece of software.
• Supports the Archive's Lucene query syntax directly (mediatype:texts AND creator:Tolstoy) for anyone who already knows it, with a plain-keyword mode for everyone else.
• Lets you filter by media type (texts, movies, audio, image, software, data, web, collection) with a single input field instead of query syntax.
• Returns a ready-to-use download URL for every item alongside the metadata, so a downstream step can fetch the file without another lookup.

Overview

For the given query (plain keyword or Lucene syntax) and optional mediaType filter, the Actor calls the Internet Archive's own public search endpoint and normalizes each result into a flat record with identifier, title, media type, description, download count, file size, language, subjects, collections, date added, item page URL, and a direct download URL. Results are capped at maxItems. A typical query returning 100-200 items finishes in a few seconds, since this is a single paginated API call, not page-by-page HTML scraping.

Supported inputs

Field	Type	Description
query	string	Required. Free-text search (matches title, description, creator) or Lucene syntax (e.g. mediatype:texts AND creator:Tolstoy)
mediaType	string	Optional filter: texts, movies, audio, image, software, etree, data, web, collection
maxItems	integer	Max records to return (1–10,000, default 200)
debug	boolean	Verbose logging

Use cases

1. Research and literature review — pull every text-mediatype item matching a research topic, with download links, to triage before downloading full files.
2. Digital preservation monitoring — track how many items under a given subject or collection exist on the Archive, and how that count changes over time.
3. Content curation for an app or newsletter — surface public-domain books, audio, or video on a theme without manually browsing archive.org.
4. Dataset building for ML or NLP — collect metadata (and direct download links) for a large set of texts or audio files matching a Lucene query, as the first step of a download pipeline.

How it works

1. The Actor builds a search request against the Internet Archive's own public search endpoint using query, with mediatype:<value> appended automatically when mediaType is set.
2. It requests the fields needed for the output schema in one paginated call, stepping through pages until maxItems is reached or results run out.
3. Each raw result is normalized into the flat schema below; fields the Archive does not have for a given item (e.g. language for a piece of software) come back as null or an empty array rather than being guessed.
4. A direct downloadUrl is built from the item's stable identifier, which works for any public item regardless of media type.

Input configuration — 3 examples

Mode 1 — plain keyword search:

{
  "query": "machine learning",
  "mediaType": "texts",
  "maxItems": 50
}

Mode 2 — Lucene query, no media type filter:

{
  "query": "mediatype:audio AND subject:jazz",
  "maxItems": 100
}

Mode 3 — collection-scoped search:

{
  "query": "collection:folkscanomy_miscellaneous AND machine learning",
  "mediaType": "texts",
  "maxItems": 20
}

Output sample

Real output from a smoke test run on query: "machine learning", mediaType: "texts", unedited (subject and collection lists trimmed for length):

{
  "identifier": "amsco-ap-world-history",
  "title": "AMSCO AP World History",
  "mediatype": "texts",
  "description": "This is the full AMSCO AP World History book.",
  "downloads": 654921,
  "itemSize": 483381567,
  "language": "eng",
  "subject": ["AMSCO AP World", "AP World", "AP World History", "AMSCO AP World History"],
  "collection": ["folkscanomy_miscellaneous", "folkscanomy", "... (300+ more favorites lists)"],
  "addedDate": "2022-04-10T18:42:16Z",
  "url": "https://archive.org/details/amsco-ap-world-history",
  "downloadUrl": "https://archive.org/download/amsco-ap-world-history"
}

{
  "identifier": "DeepLearningCollectionPDF",
  "title": "Deep Learning Collection PDF",
  "mediatype": "texts",
  "description": null,
  "downloads": 138700,
  "language": null,
  "url": "https://archive.org/details/DeepLearningCollectionPDF",
  "downloadUrl": "https://archive.org/download/DeepLearningCollectionPDF"
}

Key output fields

Field	Type	Description
identifier	string	Stable Internet Archive item identifier
title	string	Item title
mediatype	string	texts, movies, audio, software, image, etc.
description	string | null	Item description, when the uploader provided one
downloads	integer	All-time download count on the Archive
itemSize	integer | null	Total item size in bytes, when available
language	string | null	Language code, when set by the uploader
subject	array of strings	Subject/keyword tags set by the uploader
collection	array of strings	Collections the item belongs to (includes community "favorites" lists for popular items)
addedDate	string	ISO date the item was added to the Archive
url	string	Item details page on archive.org
downloadUrl	string	Direct download page for the item's files

Pricing

Pay per event, no subscription.

Event	Price	When
Actor start	$0.00005	Once per run
Item scraped	$0.0005	Per item returned (errors not charged)

Example: a run returning 200 items costs $0.00005 + 200 × $0.0005 = $0.10005.

What makes this richer than the competition

Compared with the two closest listings for the same target on the Apify Store:

• vs. maged120/archive-org-advanced-search (Internet Archive Search — Advanced Archive.org Query): that listing charges up to $0.018 per item on the Free plan (dropping only on higher paid tiers); this Actor is a flat $0.0005 per item for everyone, roughly 36x cheaper at the Free tier.
• vs. rel8ble/internet-archive-scraper (Internet Archive Scraper & API): charges a flat $0.001 per result; this Actor is half that price for the same kind of metadata plus a ready-to-use downloadUrl on every item.

Notes & limitations

• description, language, and itemSize are only as complete as what the uploader filled in on archive.org; many items legitimately have no description or language tag and will return null.
• collection can include dozens or hundreds of entries for popular items (community "favorites" lists), since the Archive treats every user favorites list as a collection; this is real Archive data, not a parsing error.
• This Actor returns metadata and a download link, not the file content itself; downloading large items (video, software ISOs) is a separate step outside this Actor.
• Very broad plain-keyword queries across all media types can return a large and noisy result set; narrowing with mediaType or Lucene syntax (e.g. subject:, creator:) gives more relevant results.

FAQ

Do I need an Internet Archive account or API key? No. The Archive's search is public; this Actor calls it the same way a logged-out visitor's browser would.

Can I search with Archive.org's own query syntax? Yes — query accepts full Lucene syntax, e.g. mediatype:texts AND creator:Tolstoy, in addition to plain keywords.

Does this download the actual files (PDFs, audio, video)? No, only metadata and a downloadUrl pointing to the item's files; fetching the file itself is a separate step.

Why do some items have null for description or language? Because the original uploader did not fill in that field on archive.org; this Actor does not invent values for missing metadata.

Can I limit results to a specific media type, like only audio? Yes, with the mediaType field, or by adding mediatype:audio directly to a Lucene query.

For AI Agents & LLM Apps

This Actor is Apify MCP-compatible and can be called directly from an LLM agent:

• Via Apify MCP (mcp.apify.com): add gio21/archive-org-items-scraper as a tool and pass {"query": "machine learning", "mediaType": "texts", "maxItems": 20} as the call arguments.
• Via API (no MCP needed):

curl -X POST "https://api.apify.com/v2/acts/gio21~archive-org-items-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "machine learning", "mediaType": "texts", "maxItems": 20}'

Disclaimer

This Actor is an independent tool that reads publicly available search results from archive.org, operated by the Internet Archive, a 501(c)(3) non-profit. It is not affiliated with, endorsed by, or sponsored by the Internet Archive. "Internet Archive" and "archive.org" are marks of the Internet Archive. Only public item metadata is collected; please consider supporting the Internet Archive directly if you rely on its collections.

SEO Keywords

archive.org scraper, internet archive scraper, archive.org search api, archive.org metadata scraper, internet archive items scraper, archive.org download link scraper, archive.org lucene query, internet archive api alternative, public domain books scraper, archive.org bulk search