Open Library Scraper · Books, Authors, Editions & Subjects

The Open Library Scraper is a high-performance HTTP actor designed to extract comprehensive book records, author profiles, publication details, edition metadata, Internet Archive ebook availability, and subjects from the Open Library API. Whether you are analyzing literary trends, indexing public domain works, enriching bibliographic databases, or tracking author publications, this scraper provides clean, structured camelCase datasets per book work item.

Maintained by reapX. Every row cites the Apify run that produced it — nothing is inferred, modelled or filled in, and a field absent from the source is absent from the row. The extracted archive for this source is browsable at reapx.dev/data/open-library-scraper/ and mirrored as an open dataset on Hugging Face and Kaggle. Questions: reapxdev@proton.me

Maintained by reapx (reapxdev@proton.me). Public data only. Every record cites a real observation from openlibrary.org.

---

Key Features

• Fast HTTP Execution: Operating directly over plain HTTP APIs without browser overhead to maximize throughput and minimize cost.
• Flexible Search Filters: Search by general keywords, subjects/genres, author names, publication year bands, or language codes.
• Pay-Per-Event Billing: Charged strictly per returned record ($0.001/record with tiered volume discounts). Zero charges on empty results or API blocks.
• Stable Addressable Identifiers: Emits clean slug identifiers (Open Library Work IDs like OL2784125W) compatible with deep entity pages.
• Rich Bibliographic Metadata: Captures titles, subtitles, primary & contributing authors, ISBNs, edition counts, fulltext scan availability, and cover image IDs.

---

⬇️ Input

The scraper accepts JSON inputs to configure search filters and item caps. All parameters are optional, but providing search criteria narrows results to your domain of interest.

Field Name	Type	Default	Description
query	String	"python"	General search query term (e.g. python, machine learning, science fiction).
subject	String	""	Filter books by subject or genre (e.g. science_fiction, history, fantasy).
author	String	""	Filter books by author name (e.g. Stephen King, Isaac Asimov, Mark Lutz).
firstPublishYear	String	""	Filter by first publication year or range (e.g. 1995 or [1990 TO 1999]).
language	String	""	Filter by language code (e.g. eng, fre, ger, spa).
maxItems	Integer	100	Maximum number of book records to extract (1 to 1,000).

Example Input Configuration

{
  "subject": "science_fiction",
  "firstPublishYear": "[1990 TO 1999]",
  "language": "eng",
  "maxItems": 100
}

---

⬆️ Output

Results are saved directly to the actor run's default dataset. Each output record represents one distinct book work object.

Dataset Schema Fields

Field Name	Type	Description	Example
slug	String	Unique Open Library Work ID without /works/ prefix.	"OL2784125W"
olKey	String	Full raw Open Library work key path.	"/works/OL2784125W"
title	String	Primary title of the book.	"Learning Python"
subtitle	String	Subtitle of the book if available.	"Powerful Object-Oriented Programming"
authorName	String	Primary author name.	"Mark Lutz"
authorNames	Array	List of all contributing author names.	["Mark Lutz"]
authorKeys	Array	Open Library author keys (OLIDs).	["OL22643A"]
firstPublishYear	Integer	Year the work was first published.	1999
editionCount	Integer	Total number of known editions on Open Library.	15
hasFulltext	Boolean	Whether full text is readable/borrowable on Internet Archive.	true
language	Array	Language codes for available editions.	["eng"]
subject	Array	Tagged subjects and genres.	["Python (Computer program language)"]
publisher	Array	Publishers associated with the work.	["O'Reilly Media"]
isbn	Array	List of ISBN-10 and ISBN-13 strings.	["0596158106"]
coverEditionKey	String	Key of the edition used for the cover.	"OL24213197M"
coverI	Integer	Numeric ID for Open Library Covers API.	6496660
ebookAccess	String	Digital access status (e.g. borrowable, printdisabled).	"borrowable"
ia	Array	Internet Archive item identifiers.	["learningpython0000lutz"]
publicScanB	Boolean	Whether a public scan is accessible.	true

Sample Output Item

{
  "slug": "OL2784125W",
  "olKey": "/works/OL2784125W",
  "title": "Learning Python",
  "subtitle": "Powerful Object-Oriented Programming",
  "authorName": "Mark Lutz",
  "authorNames": [
    "Mark Lutz"
  ],
  "authorKeys": [
    "OL22643A"
  ],
  "firstPublishYear": 1999,
  "editionCount": 15,
  "hasFulltext": true,
  "language": [
    "eng"
  ],
  "subject": [
    "Python (Computer program language)",
    "Computer programming",
    "Object-oriented programming (Computer science)"
  ],
  "publisher": [
    "O'Reilly Media"
  ],
  "isbn": [
    "0596158106",
    "9780596158101"
  ],
  "coverEditionKey": "OL24213197M",
  "coverI": 6496660,
  "ebookAccess": "borrowable",
  "ia": [
    "learningpython0000lutz"
  ],
  "publicScanB": true
}

---

How it works

1. Parameter Construction: The actor builds targeted query strings for Open Library's Search API based on your configured query, subject, author, firstPublishYear, and language.
2. Paginated Retrieval: Requests are dispatched sequentially over HTTP with headers configured for optimal performance and rate-limit compliance.
3. Resilient HTTP Backoff: Automatic retries with exponential backoff handle transient network errors or HTTP 429 rate limit responses gracefully.
4. Data Normalization: Raw API documents are parsed, cleaned, and mapped to addressable slug keys and camelCase properties.
5. Streaming Dataset Output: Records are pushed to Apify's default dataset progressively as pages complete, ensuring data is saved incrementally.

---

Code Examples

Integration with Python SDK

from apify_client import ApifyClient

# Initialize the client with your Apify API token
client = ApifyClient("YOUR_APIFY_TOKEN")

# Prepare actor input
run_input = {
    "subject": "science_fiction",
    "firstPublishYear": "[1990 TO 1999]",
    "maxItems": 50
}

# Run the actor and wait for completion
run = client.actor("reapx/open-library-scraper").call(run_input=run_input)

# Fetch dataset items
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Book: {item.get('title')} by {item.get('authorName')} (OLID: {item.get('slug')})")

Integration with Node.js SDK

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({
    token: 'YOUR_APIFY_TOKEN',
});

const input = {
    author: 'Stephen King',
    maxItems: 50
};

(async () => {
    const run = await client.actor('reapx/open-library-scraper').call(input);
    const { items } = await client.dataset(run.defaultDatasetId).listItems();
    items.forEach((item) => {
        console.log(`[${item.slug}] ${item.title} - ${item.firstPublishYear}`);
    });
})();

---

❓ FAQ

1. How is this scraper billed?

This actor uses Pay-Per-Event (PPE) pricing. You pay $0.001 per returned book record written to your dataset. Tiered volume discounts automatically apply for higher usage volumes (down to $0.0004/record on Diamond tier). If an API query yields no results or fails, you pay $0.

2. Can I filter by specific subjects or genres?

Yes. Use the subject input field with standard Open Library subject slugs such as science_fiction, history, fantasy, biography, computer_science, or art.

3. What is the difference between slug and olKey?

slug is the stripped work identifier (e.g. OL2784125W), which serves as a clean, stable addressable entity key. olKey is the original raw path returned by Open Library (e.g. /works/OL2784125W).

4. Are fulltext scans available for all returned books?

hasFulltext indicates whether Internet Archive holds a digitized scan. The ia field provides the specific Internet Archive identifier(s) when available.

---

💬 Your feedback

We continuously improve our scrapers based on user feedback. If you have feature requests, bug reports, or questions, contact us at reapxdev@proton.me.

---

Disclaimer: Unofficial - not affiliated with Open Library or Internet Archive. Collects public data only. reapx. Contact reapxdev@proton.me.

🧪 Example input

A real, runnable configuration — this is an actual input this Actor has run with.

{
  "subject": "science_fiction",
  "maxItems": 100
}

📄 Sample output

One real row from a real run of this Actor, unedited.

{
  "slug": "OL2784125W",
  "olKey": "/works/OL2784125W",
  "title": "Learning Python",
  "subtitle": null,
  "authorName": "Mark Lutz",
  "authorNames": [
    "Mark Lutz",
    "David Ascher"
  ],
  "authorKeys": [
    "OL411267A",
    "OL2726848A"
  ],
  "firstPublishYear": 1999,
  "editionCount": 23,
  "hasFulltext": true,
  "language": [
    "ger",
    "eng"
  ],
  "subject": null,
  "publisher": null,
  "isbn": null,
  "coverEditionKey": "OL9497269M",
  "coverI": 1312568,
  "ebookAccess": "borrowable",
  "ia": [
    "einfuhrunginpyth0000lutz",
    "learningpython00lutz",
    "learningpythonth00lutz"
  ],
  "publicScanB": false
}

⚠️ Run outcomes and error handling

This Actor reports what happened in the run's status message, and it always keeps whatever it collected. These are the outcomes you can get and what each one means.

Outcome	What it means
Success	Rows were returned and you were charged record-returned at $0.001 per row.
No matches	The source returned nothing for your filters. Nothing is charged. Widen the date window or drop a filter.

What is guaranteed either way

• Every row is pushed as it is built, not buffered to the end of the run. Anything that buffers output loses everything to a timeout, a block or a migration; this does not.
• A field absent from the source is absent from the row. Nothing is inferred, modelled or filled in to make a row look complete.