Goodreads Review Scraper
Pricing
from $1.21 / 1,000 results
Goodreads Review Scraper collects public reviews from Goodreads book URLs or IDs: star rating, review text, reviewer profile, likes, comments, shelves and timestamps. Filter by language and sort order. No login or cookies. Export as JSON, CSV or Excel, or use the API.
All notable changes to this Actor are documented here.
Code/43 - Goodreads Review Scraper)The reference scraper this Actor was originally built from has since been rewritten
(confirmed live, 2026-10-15): Goodreads removed the static production AppSync
apiKey from its public JS bundle, so the old DEFAULT_API_KEY +
getBookByLegacyId discovery path this Actor relied on (added in [1.1]) is no
longer the live mechanism Goodreads uses, even though the specific key value
happened to still authenticate at the time of this rebuild. The reference now
reads the live key and both work/book resource ids directly out of each
book's own /reviews page on every run — no hardcoded key, no separate
resource-id lookup call, and no stale-key maintenance going forward. This
session ported that discovery pipeline, verified end-to-end against the live
site (resource id extraction, live key extraction, review-service discovery,
and a real paginated review fetch all confirmed working), and re-platformed
its retry/pause/connection-escalation architecture onto Apify's own proxy
tiers instead of a single external one.
src/network.py (new, replaces src/helper.py) — live discovery per book
(no hardcoded or cached key), quick-retry + pause-and-retry rounds, and
connection escalation via Actor.create_proxy_configuration (direct →
user-configured Apify proxy → residential), replacing the previous
aiohttp + hardcoded-key-then-page-scrape-fallback design.src/parser.py (new, replaces src/review_extractor.py) — pure parsing
ported from the current reference, with two data-loss bugs fixed: the
reference's own GraphQL query fetches preReleaseBookSource and five
extra shelf fields (displayName, editable, default, actionType,
sortOrder) but drops all of them when building its own output row
(confirmed against its own output.json sample and against a live
request in this session) — all six are now included. viewerHasLiked
and creator.viewerRelationshipStatus are also carried through (always
null for this Actor's logged-out requests, which is Goodreads' real
answer, not a placeholder). The two *Iso timestamp fields added in an
earlier version of this Actor are kept as an additive bonus beyond the
reference's own output shape.src/free_limits.py (new) — free-plan usage limits (20 reviews per book
per run, 200 per month across all runs), counted in a named key-value
store (free-quota) in the running user's own account. Unlimited for
local runs, Console test runs, and paying users.src/main.py — input field names now match the reference exactly:
urls → bookUrls, the nested filtersAndOptions.{sortBy,languageCode, reviewEdition}sortBy, languageCode, Review_Edition.
Bare numeric book ids are still accepted and rewritten automatically
(kept from the previous build; the reference itself does not support
this). Books are now processed with up to 4 running concurrently
(previously strictly sequential). Every dataset push now follows the
mandatory charged_event_name="row_result" + event_charge_limit_reached
pattern exactly; the previous build's "disable charging forever after
the first error" fallback is removed as unnecessary — pushes already
succeed and save data even when PPE charging isn't configured for a run.
maxItems=0 (or negative) now fails the run with a clear validation
error instead of silently falling back to a default..actor/input_schema.json, .actor/dataset_schema.json,
.actor/output_schema.json — updated for the new field names and the
restored output fields; proxyConfiguration now actually feeds
Actor.create_proxy_configuration(actor_proxy_input=...) (previously
present in the form but not wired into the proxy logic at all).Dockerfile — added ENV PYTHONUNBUFFERED=1 so log lines reach the
Apify Console reliably during a real run.requirements.txt — added curl_cffi (browser TLS/HTTP impersonation,
replacing the removed aiohttp-only HTTP layer) and apify-client;
raised the apify floor to >=2.3.0 for charged_event_name support.shelf fields confirmed non-null in a live response
(displayName, editable, default, actionType, sortOrder).bookUrls: ["26032825"], maxItems: 3,
sortBy: "newest", languageCode: "en", Review_Edition: "only_this_book"Before changing anything, this session re-verified the whole pipeline against the real, current Goodreads site and AWS AppSync endpoint, live, rather than trusting the previous fix:
GET https://www.goodreads.com/book/show/<id> → HTTP 200, and it (and,
today, even the /reviews-suffixed page — the WAF challenge from [1.1] is
currently not being served to this Actor's outbound IPs) contains the
current inline "apiKey":"da2-d2fyuybwsbf3poyquvbp2mbiwu".getBookByLegacyId,
still returns HTTP 200 with real book/work resource ids — the
DEFAULT_API_KEY from [1.1] has not rotated again yet.testt0 account, actor
PyGpADimS6A9fJiQb), each against a different book, both finished
SUCCEEDED with real, non-empty review rows (rating, text, reviewer,
likes/comments) on the plain direct connection tier (no proxy escalation
needed) — see "Verified live" below for run ids and samples.So the GraphQL/key-rotation self-healing itself is intact today. But the task was to find why this Actor "keeps needing repeated maintenance" despite that fix, so this session went further and exercised failure paths the [1.1] session hadn't: a book id list mixing a working book with one Goodreads can't resolve.
Evidence. Running the Actor live with two book URLs — one a nonexistent
(but otherwise well-formed) book id, one a real book — produced a FAILED
run with zero dataset rows, even though the second, real book would have
returned reviews fine on its own (confirmed separately). The traceback showed
the first book's failure propagating all the way out of main() and
crashing the whole Actor:
Two compounding bugs caused this:
src/main.py's book loop had no per-book error handling. Every URL in
urls was processed by one unguarded await fetch_reviews_for_book(...)
call inside the for loop in main(). Any book that ultimately failed
(wrong/rotated id, Goodreads returning an error for that specific book,
etc.) raised straight out of the loop, aborting every subsequent book in
the list and ending the run FAILED with whatever had (or, if it was the
first book, hadn't yet) been pushed. For any multi-URL input, a single
bad entry was enough to zero out the whole run — a very plausible
explanation for intermittent "keeps failing to produce output" reports
whenever a user's list included one stale/mistyped id.
src/helper.py classified every GraphQL error the same way — as a
possible-bad-apiKey "block" (GoodreadsBlocked), worth a key refresh, 3
retries, and two more full retry rounds through datacenter and
residential proxy escalation. Confirmed live against the real AppSync
endpoint that this conflates two very different, and very differently
actionable, situations:
legacyId → HTTP 200 with
"message": "Variable 'legacyId' has an invalid value." (no
errorType).2100000000) → HTTP 200
with "errorType":"RESOURCE_NOT_FOUND",
"message":"Book not found (maybe deleted, and no replacement exists).".Neither is a block or a rejected key — Goodreads answered cleanly and
definitively both times. But the old code retried each one 3× per proxy
tier × 3 tiers before finally giving up, live-timed at ~68 seconds to
fail a single already-known-unresolvable book (see the 2n5Nbok5fEVaAPD2Q
run below) — wasting most of a short run's timeout/runtime on a foregone
conclusion, and, combined with bug 1, taking any other books in the list
down with it.
src/helper.py — added GoodreadsBookNotFound(RuntimeError), a new,
distinct exception for a confirmed, well-formed "no such book" GraphQL
response (any error without an auth-related marker in its errorType or
message — unauthorized, api key/apikey, forbidden,
accessdenied). _resolve_resource_ids_via_graphql() now only raises the
retry/escalate-worthy GoodreadsBlocked when an error actually looks
auth-related (or when the response has neither a book nor any error at all,
which still smells like a silently-rejected key); every other clean error
shape raises GoodreadsBookNotFound instead.src/main.py — discover_api_config_with_retry() now re-raises
GoodreadsBookNotFound immediately instead of retrying it 3×;
_should_escalate() now returns False for it instead of triggering
proxy-tier escalation. A confirmed "no such book" answer fails in well
under a second instead of ~68 seconds.src/main.py — main()'s per-book loop now wraps each
fetch_reviews_for_book() call in try/except: a failing book is logged
with a clear ❌ Skipping book i/n (...) warning and the run moves on to
the next URL instead of aborting. All per-book failures are summarized in
one warning block at the end. The run only raises (and fails) if zero
rows were collected across all requested books — a partial list of
good books now still produces a SUCCEEDED run with their real data, while
an all-bad list still fails clearly and fast instead of silently returning
an empty, green dataset.GET .../book/show/26032825 → HTTP 200; extracted
apiKey=da2-d2fyuybwsbf3poyquvbp2mbiwu; POSTed to AppSync getBookByLegacyId
→ HTTP 200, real ids for The Cruel Prince.getBookByLegacyId for a valid-range but nonexistent id (2100000000) →
HTTP 200, errorType:"RESOURCE_NOT_FOUND" (confirms the new
GoodreadsBookNotFound classification is correct, not a block).2n5Nbok5fEVaAPD2Q
(urls: 999999999999, 1885) → FAILED, 0 dataset rows, ~68s spent
retrying/escalating (direct → datacenter → residential) a book id that was
never going to resolve, before crashing before the second (valid) book was
ever attempted.3ljZGXNB7RbaG5KGB
(urls: 2100000000 [bad], 1885 [good]) → SUCCEEDED in ~3s total; bad
book skipped in ~0.1s with a clear ❌/⚠️ log, good book still saved 3
real reviews to the dataset (eMRtevpnGGTceh3eL, itemCount 3).hHMeTqm45SzwuvaAs
(urls: 2100000000) → FAILED in <1s with a clear
RuntimeError: No reviews were collected for any of the 1 requested book(s)...SUCCEEDED with real data:
aEZzBZ3ssUGesRjQe — The Cruel Prince (id 26032825), 5/5 rows saved
(dataset T5GfjVbwQNCLgLpAE), e.g. reviewer "Melanie (meltotheany)",
5★, likeCount 3335.03OJLNh8i0UV0WDde — The Hobbit (id 5907), sorted newest, 5/5 rows
saved (dataset qCzgNvwkif7gtL5pO), e.g. reviewer "Michelle", 5★, text
"my precious".testt0, CUSTOM/BRONZE) does have
working RESIDENTIAL proxy group access (tested a direct proxied request
through groups-RESIDENTIAL — got a real residential exit IP), despite the
account API listing availableCount: 0 for that group — so
ProxyState's residential-escalation tier is not a dead end on this
account. Residential bandwidth is capped at 10 GB/month on this plan; if a
future run pattern relies on residential escalation heavily, that quota is
worth monitoring.Root cause. The Actor authenticates to Goodreads' public GraphQL (AWS AppSync)
API with a hardcoded x-api-key. Goodreads has since rotated that key —
da2-xpgsdydkbregjhpr6ejzqdhuwy (the old hardcoded value) now returns
HTTP 401 UnauthorizedException from AppSync, confirmed live against
https://kxbwmqov6jgg3daaamb744ycu4.appsync-api.us-east-1.amazonaws.com/graphql.
The Actor already had a designed fallback for exactly this ("refresh the key from
the page") but that fallback was itself broken in two independent ways, so every
run fell straight through to a hard failure instead of self-healing:
src/main.py built the fallback's target URL with to_reviews_page_url(),
i.e. the /book/show/<id>/reviews page. That page is served behind Goodreads'
AWS WAF JavaScript challenge and returns HTTP 202 (confirmed live) — so the
fallback always tried to scrape a page it could never actually load./book/show/<id> page (no
/reviews suffix — confirmed live to return HTTP 200 with full HTML),
extract_api_data() in src/helper.py only looked inside external .js
bundle files for a "Production":{"apiKey":"...","endpoint":"https://..."}
pattern. Goodreads no longer ships the AppSync config that way at all: none
of the 26 JS chunks referenced from a live book page contain the strings
apiKey, appsync, or da2- any more. The key is now inlined directly in
the book page's own server-rendered HTML (Next.js SSR props), as
"dataSource":"Production","apiKey":"da2-...", with no adjacent
"endpoint" field at all — confirmed live (zero occurrences of the string
endpoint anywhere in the page). The old regex, which required an
"endpoint" key inside the same JS blob, could never match this shape.src/helper.py — discover_goodreads_api_config() now always rebuilds the
fallback's target as the plain f"{BASE_URL}/book/show/{legacy_id}" URL from
the parsed numeric book id, instead of trusting whatever URL shape the caller
passed in. This guarantees the fallback never targets the WAF-challenged
/reviews page again, regardless of how it's called.src/helper.py — _refresh_api_config_from_page() now runs extract_api_data()
against the fetched book page's own HTML first (that's where the key actually
lives now), before falling back to fetching and scanning its JS bundles. This
also cuts ~26 unnecessary HTTP requests per key refresh in the common case.src/helper.py — extract_api_data() gained a second match path: a bare
"apiKey":"da2-..." pattern with no required adjacent "endpoint", paired
with the still-stable DEFAULT_ENDPOINT constant (the AppSync endpoint host
itself has not changed — only the key format/location did). The original
endpoint-paired regex is kept first, so a future reversion to the old bundle
format would still be picked up.src/helper.py — DEFAULT_API_KEY updated to the currently valid key
observed live (da2-d2fyuybwsbf3poyquvbp2mbiwu), confirmed working end-to-end
against getBookByLegacyId for a real book id. This is a best-effort fast
path only; the fallback above is what keeps the Actor working through the
next rotation without a code change.HTTP 401 UnauthorizedExceptionGET https://www.goodreads.com/book/show/<id>/reviews → HTTP 202, empty
body (confirmed the WAF challenge on the reviews page).GET https://www.goodreads.com/book/show/<id> (no /reviews) → HTTP 200,
full HTML containing the live apiKey inline.getBookByLegacyId → HTTP 200 with real book/work resource ids
(The Cruel Prince, book id 26032825).Initial version.