OpenLibrary Books Scraper

Book catalogue data from the Internet Archive's OpenLibrary via its own public JSON API: full-text search, subject/genre browsing, and rich per-work detail.

---

What you get

searchQueries — one SEARCH_SUMMARY + N BOOK rows per query. Each BOOK row carries title, author(s), first-publish year, edition count, language, publisher, ISBNs, ebook access (via Internet Archive), subject tags, ratings, reading-log counts, and cross-site identifiers (Goodreads, Amazon, LibraryThing, ISFDB) — see searchFields to customize which fields come back.

subjectQueries — same shape, browsing by subject/genre slug (e.g. science_fiction, fantasy) instead of free text.

workLookups — one WORK row per OpenLibrary work id: full description, complete subject list, and linked author keys — richer than what a search/subject row carries, at the cost of one request per id (not auto-attached to search results, see CRAWLING_METHOD.md §6).

Both searchQueries and subjectQueries use REAL pagination — unlike some other targets in this portfolio, OpenLibrary's page/offset parameters genuinely advance (verified: zero overlap between pages), so this actor pages through results up to maxItemsPerQuery rather than a single fetch.

---

Input

{
  "searchQueries": ["dune frank herbert"],
  "subjectQueries": ["science_fiction"],
  "workLookups": ["OL893414W"],
  "maxItemsPerQuery": 50
}

• searchQueries — free-text catalogue search.
• subjectQueries — subject/genre slug browse (find slugs at openlibrary.org/subjects).
• workLookups — full work detail by id (OL893414W, with or without a /works/ prefix or .json suffix).
• searchFields — advanced, optional: override the default curated field set with a custom list, or ["*"] for upstream's raw firehose (adds ~50 trending-score columns — see CRAWLING_METHOD.md §5).
• maxItemsPerQuery — caps rows per search/subject query (0 = no explicit cap, still bounded by a safety net). Doesn't apply to workLookups (always exactly one row per id).

---

Known limits

1. searchFields only affects searchQueries. Subject-browse rows and work-detail rows use whatever fields OpenLibrary's own endpoints return by default for those surfaces — there's no equivalent field filter on /subjects/<slug>.json or /works/<id>.json.

2. Pagination is real but not perfectly stable at the page boundary. Relevance-sorted results with tied scores can appear on two consecutive pages of the same query (confirmed: 20 of 120 rows in one test). This actor dedupes by key across the whole query automatically — you'll never see a duplicate row in the output — and reports how many were caught via duplicatesSkipped on the SEARCH_SUMMARY row.

3. Stale work ids redirect once, not indefinitely. A small number of workLookups ids are old/merged records that redirect to a canonical id — followed automatically (see _redirectedFrom on the resulting row). A second redirect in a row is reported as an unexpected_shape ERROR row rather than chased further.

3. Default pacing is 10 seconds between requests, not a technical minimum — it directly honors openlibrary.org's own robots.txt Crawl-delay: 10 set specifically for ClaudeBot/anthropic-ai. Lower minRequestInterval if that policy signal doesn't apply to your use case; no WAF or rate-limit was observed technically.

---

Errors are honest, no disguised-200 traps

A bad searchQueries/subjectQueries entry gets an honest empty SEARCH_SUMMARY (upstreamTotal: 0), never a silently-widened catalogue. A bad workLookups id gets a genuine, well-formed 404 → ERROR row with _error: "not_found".

Anti-bot posture

None encountered. 5/5 curl_cffi TLS profiles clean 200, byte- identical response sizes across profiles.

Policy

https://openlibrary.org/robots.txt explicitly names BOTH anthropic-ai and ClaudeBot with only a Crawl-delay: 10 — no Disallow rules of their own, meaning the wildcard group's restrictive list (/api, /search*, etc.) does not apply to this actor's identity. The most deliberately permissive policy signal found in this portfolio to date.