Internet Archive Search Scraper

Search archive.org and export item metadata -- books, audio recordings, videos, software, images, and web captures -- using the official Advanced Search API. No API key or browser required.

What it scrapes

The actor queries the Internet Archive's public JSON search endpoint and fetches item metadata. Optionally, it fetches the file manifest for each item (one extra request per item). Results are exported to a dataset.

Inputs

Name	Type	Default	Description
query	string	mediatype:texts AND subject:python	Archive.org search query (Lucene syntax). Supports field-scoped queries: creator:tolkien, subject:jazz, collection:librivoxaudio.
mediatype	string	(empty)	Narrow to a media type: texts, audio, movies, software, image, web, data.
sort	string	downloads desc	Sort order. Examples: date desc, titleSorter asc, avg_rating desc.
maxItems	integer	100	Maximum items to return. Hard cap: 10,000.
includeFiles	boolean	false	Also fetch the file manifest for each item. Makes one extra request per item -- significantly slower for large result sets.

Query syntax

Archive.org uses Lucene syntax. Some useful patterns:

subject:jazz AND mediatype:audio
creator:"Ernest Hemingway"
collection:gutenberg AND language:english
date:[2020-01-01 TO 2023-12-31]

Output fields

Field	Type	Description
identifier	string	Unique Archive.org item ID
title	string	Item title
creator	array	Author(s) or creator(s)
description	string	Item description
mediatype	string	texts, audio, movies, software, image, etc.
subject	array	Subject tags
date	string	Original publication date
year	integer	Publication year
language	array	Language(s)
downloads	integer	Total download count
item_size	integer	Total size in bytes
collection	array	Collections the item belongs to
publicdate	string	Date added to Archive.org (ISO 8601)
item_url	string	Direct URL to the item page
files	array	File manifest (only when includeFiles: true)

Sample output

Result from query mediatype:texts AND subject:python, sorted by downloads desc:

{"identifier":"python_ebooks_2020","title":"Python Ebooks","creator":["strugglez"],"description":"Python Ebooks","mediatype":"texts","subject":["python"],"date":null,"year":null,"language":["Latin"],"downloads":257128,"item_size":49856126105,"collection":["booksbylanguage_latin","booksbylanguage"],"publicdate":"2020-09-27T09:24:26Z","item_url":"https://archive.org/details/python_ebooks_2020","files":null}

Pricing

Pay-per-result: $0.50 per 1,000 items. Fetching 100 items costs $0.05. Enabling includeFiles does not increase the per-item charge but increases runtime.

Limitations

• Archive.org caps search results at 10,000 items per query. For larger data needs, paginate using date ranges or subject filters.
• includeFiles: true makes one extra HTTP request per item -- 1,000 items takes significantly longer and may hit rate limits.
• Some fields (date, year, language) are not consistently filled by archive contributors.
• The collection array includes personal favorites (prefixed fav-) alongside curated collections.

Rate limits

The actor uses 500ms delays between search pages and 600ms delays between per-item file requests. Concurrency is kept at 1 (sequential). This respects archive.org's infrastructure.

Legal

Archive.org's public search API is openly accessible with no authentication. Only public metadata is exported. No paywalled content or login-restricted data is accessed.