# OpenLibrary Books Scraper (`scrapyx/openlibrary-books-scraper`) Actor

Book catalogue data from the Internet Archive's OpenLibrary: full-text search, subject/genre browsing, and rich work detail (description, subjects, linked authors), with identifiers, ratings and ebook availability via its own public API.

- **URL**: https://apify.com/scrapyx/openlibrary-books-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.75 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## OpenLibrary Books Scraper

Book catalogue data from the Internet Archive's [OpenLibrary](https://openlibrary.org)
via its own public JSON API: full-text search, subject/genre browsing,
and rich per-work detail.

***

### What you get

**`searchQueries`** — one `SEARCH_SUMMARY` + N `BOOK` rows per query.
Each `BOOK` row carries title, author(s), first-publish year, edition
count, language, publisher, ISBNs, ebook access (via Internet Archive),
subject tags, ratings, reading-log counts, and cross-site identifiers
(Goodreads, Amazon, LibraryThing, ISFDB) — see `searchFields` to
customize which fields come back.

**`subjectQueries`** — same shape, browsing by subject/genre slug (e.g.
`science_fiction`, `fantasy`) instead of free text.

**`workLookups`** — one `WORK` row per OpenLibrary work id: full
description, complete subject list, and linked author keys — richer
than what a search/subject row carries, at the cost of one request per
id (not auto-attached to search results, see `CRAWLING_METHOD.md` §6).

Both `searchQueries` and `subjectQueries` use REAL pagination — unlike
some other targets in this portfolio, OpenLibrary's `page`/`offset`
parameters genuinely advance (verified: zero overlap between pages), so
this actor pages through results up to `maxItemsPerQuery` rather than a
single fetch.

***

### Input

```jsonc
{
  "searchQueries": ["dune frank herbert"],
  "subjectQueries": ["science_fiction"],
  "workLookups": ["OL893414W"],
  "maxItemsPerQuery": 50
}
```

- **`searchQueries`** — free-text catalogue search.
- **`subjectQueries`** — subject/genre slug browse (find slugs at
  `openlibrary.org/subjects`).
- **`workLookups`** — full work detail by id (`OL893414W`, with or
  without a `/works/` prefix or `.json` suffix).
- **`searchFields`** — advanced, optional: override the default curated
  field set with a custom list, or `["*"]` for upstream's raw firehose
  (adds ~50 trending-score columns — see `CRAWLING_METHOD.md` §5).
- **`maxItemsPerQuery`** — caps rows per search/subject query (0 = no
  explicit cap, still bounded by a safety net). Doesn't apply to
  `workLookups` (always exactly one row per id).

***

### Known limits

**1. `searchFields` only affects `searchQueries`.** Subject-browse rows
and work-detail rows use whatever fields OpenLibrary's own endpoints
return by default for those surfaces — there's no equivalent field
filter on `/subjects/<slug>.json` or `/works/<id>.json`.

**2. Pagination is real but not perfectly stable at the page boundary.**
Relevance-sorted results with tied scores can appear on two consecutive
pages of the same query (confirmed: 20 of 120 rows in one test). This
actor dedupes by key across the whole query automatically — you'll never
see a duplicate row in the output — and reports how many were caught via
`duplicatesSkipped` on the `SEARCH_SUMMARY` row.

**3. Stale work ids redirect once, not indefinitely.** A small number of
`workLookups` ids are old/merged records that redirect to a canonical id
— followed automatically (see `_redirectedFrom` on the resulting row). A
second redirect in a row is reported as an `unexpected_shape` `ERROR`
row rather than chased further.

**3. Default pacing is 10 seconds between requests**, not a technical
minimum — it directly honors openlibrary.org's own `robots.txt`
`Crawl-delay: 10` set specifically for `ClaudeBot`/`anthropic-ai`.
Lower `minRequestInterval` if that policy signal doesn't apply to your
use case; no WAF or rate-limit was observed technically.

***

### Errors are honest, no disguised-200 traps

A bad `searchQueries`/`subjectQueries` entry gets an honest empty
`SEARCH_SUMMARY` (`upstreamTotal: 0`), never a silently-widened
catalogue. A bad `workLookups` id gets a genuine, well-formed 404 →
`ERROR` row with `_error: "not_found"`.

### Anti-bot posture

**None encountered.** 5/5 `curl_cffi` TLS profiles clean 200, byte-
identical response sizes across profiles.

### Policy

`https://openlibrary.org/robots.txt` explicitly names BOTH
`anthropic-ai` and `ClaudeBot` with only a `Crawl-delay: 10` — no
Disallow rules of their own, meaning the wildcard group's restrictive
list (`/api`, `/search*`, etc.) does not apply to this actor's identity.
The most deliberately permissive policy signal found in this portfolio
to date.

# Actor input Schema

## `searchQueries` (type: `array`):

Full-text catalogue search, e.g. 'dune frank herbert'. Each query gets its own SEARCH\_SUMMARY + BOOK rows. Pagination is REAL on this endpoint (verified: no overlap between pages), so this actor pages through results up to 'maxItemsPerQuery' rather than a single fetch. A nonsense query returns an honest empty result (numFound: 0), never a silently-widened catalogue.

## `subjectQueries` (type: `array`):

Browse works tagged with a subject, e.g. 'science\_fiction', 'fantasy', 'poetry'. Slugs are OpenLibrary's own (lowercase, spaces as underscores) -- browse https://openlibrary.org/subjects to find one. Each entry gets its own SEARCH\_SUMMARY + BOOK rows, paged the same way as search. An unrecognised slug returns an honest work\_count: 0, not a fallback to a different subject.

## `workLookups` (type: `array`):

Full detail for one work by its OpenLibrary id (e.g. 'OL893414W', with or without a '/works/' prefix or '.json' suffix -- all accepted). Adds a full description, complete subject list, and linked author keys beyond what search/subject rows carry. A bad id is reported as a clean not\_found ERROR row. Some ids are stale/merged and redirect to a canonical id -- followed automatically once, noted via '\_redirectedFrom'.

## `searchFields` (type: `array`):

Override which fields the search API returns per BOOK row (only affects 'searchQueries', not 'subjectQueries' or 'workLookups'). Leave empty for this actor's curated default field set (identifiers, ratings, ebook availability -- see README). Pass \['\*'] for upstream's full raw field set, which additionally includes ~50 hour-of-day/day-of-week trending-score columns.

## `maxItemsPerQuery` (type: `integer`):

Caps rows per search/subject query, paging as needed to reach it (up to 100 rows per request, OpenLibrary's own practical per-request cap). 0 = no explicit cap (still bounded by a 50-page safety net). Does not apply to 'workLookups' (always exactly one row per id).

## `maxConcurrency` (type: `integer`):

How many requests may be in flight at once. This absorbs latency variance; it is NOT the throttle — the crawl rate is set by 'Minimum interval between requests' below.

## `minRequestInterval` (type: `number`):

The actual speed control: the shortest gap between two requests, across all workers. openlibrary.org's own robots.txt sets a 10-second crawl-delay specifically for the ClaudeBot/anthropic-ai groups (not a technical WAF limit -- no WAF was observed); this default of 10 honors that policy signal directly rather than just the minimum a rate-limiter would force.

## `proxyConfiguration` (type: `object`):

Residential with no country pin is the cloud default. No WAF was observed during testing (5/5 TLS profiles clean, byte-identical responses), and this actor's identity is explicitly welcomed by openlibrary.org's own robots.txt.

## Actor input object example

```json
{
  "searchQueries": [
    "dune frank herbert"
  ],
  "subjectQueries": [
    "science_fiction"
  ],
  "workLookups": [],
  "searchFields": [],
  "maxItemsPerQuery": 50,
  "maxConcurrency": 5,
  "minRequestInterval": 10,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "dune frank herbert"
    ],
    "subjectQueries": [
        "science_fiction"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/openlibrary-books-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["dune frank herbert"],
    "subjectQueries": ["science_fiction"],
}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/openlibrary-books-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "dune frank herbert"
  ],
  "subjectQueries": [
    "science_fiction"
  ]
}' |
apify call scrapyx/openlibrary-books-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/openlibrary-books-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qqYZQyKGEQSMqyf2T/builds/DKxbrxZBEcXyPaSAG/openapi.json