Internet Archive Scraper

Extract structured records from the Internet Archive for research, discovery, and bulk data export.

Use it to search archive.org by keyword or open a direct item/search URL and get clean dataset rows with optional full item metadata.

Pricing

This Actor uses Pay-Per-Event pricing. Each record written to the default dataset triggers one dataset-item event at USD 0.0025 per item (USD 2.50 per 1,000 items).

Platform usage costs are passed through separately and remain enabled. The minimum maximum total charge is USD 2.00. When a run reaches its charge limit, the current successfully stored item is included and the Actor stops further requests and dataset writes.

What this Actor does

• Searches the Internet Archive public advanced search API.
• Supports keyword search, Lucene-style filters, and direct startUrls.
• Extracts normalized item fields ready for CSV, JSON, or downstream AI workflows.
• Optionally fetches full item metadata and file lists for each result.
• Handles pagination until the requested limit or end of results.

Best for

• Digital humanities and archival research
• Media discovery and catalog exports
• Dataset building for AI and RAG pipelines
• Monitoring collections, creators, or topic-based archive search results

Input example

{
  "query": "frankenstein",
  "mediaType": "texts",
  "sortBy": "relevance",
  "maxItems": 50,
  "fetchItemDetails": false,
  "startUrls": []
}

Direct URL example

{
  "startUrls": [
    { "url": "https://archive.org/details/frankenstein00shel" }
  ],
  "maxItems": 1,
  "fetchItemDetails": true
}

Output sample

{
  "identifier": "frankenstein00shel",
  "title": "Frankenstein",
  "url": "https://archive.org/details/frankenstein00shel",
  "creator": "Shelley, Mary",
  "date": "2022-01-01T00:00:00Z",
  "mediatype": "texts",
  "description": "Classic horror novel",
  "subject": ["Fiction", "Horror"],
  "language": "eng",
  "downloads": 1234,
  "collection": ["gutenberg", "opensource"],
  "licenseurl": "https://creativecommons.org/licenses/by/4.0/",
  "publicdate": "2020-01-01T00:00:00Z",
  "item_size": 1048576,
  "year": 2022,
  "format": ["PDF", "EPUB"],
  "files": null,
  "detailMetadata": null
}

AI & LLM integration

The Actor outputs normalized JSON records with stable fields that are easy to index, summarize, cluster, or enrich.

Use the dataset for:

• RAG corpora built from archive metadata
• Topic and collection discovery
• Research automation and catalog QA
• Bulk exports into BI tools, vector databases, or ETL pipelines

Notes

• maxItems = 0 means unlimited scraping until the API result set is exhausted.
• fetchItemDetails adds full metadata requests and slows the run.
• Search queries can use Internet Archive Lucene syntax.
• When a search URL is provided, the Actor applies the configured media type filter.
• Every successful output record is written once to the default dataset and billed once as dataset-item.