Goodreads Review Scraper: Extract book reviews, ratings, reviewer names, dates, review text, book details, and engagement data. Analyze reader sentiment, recurring feedback, popular themes, and audience preferences for book research, market analysis, and content insights.
All notable changes to this Actor are documented here.
[1.2] - 2026-09-07
Verified: the [1.1] key-rotation fix is still working live
Before changing anything, this session re-verified the whole pipeline against
the real, current Goodreads site and AWS AppSync endpoint, live, rather than
trusting the previous fix:
GET https://www.goodreads.com/book/show/<id> → HTTP 200, and it (and,
today, even the /reviews-suffixed page — the WAF challenge from [1.1] is
currently not being served to this Actor's outbound IPs) contains the
current inline "apiKey":"da2-d2fyuybwsbf3poyquvbp2mbiwu".
That exact key, POSTed to the AppSync endpoint for getBookByLegacyId,
still returns HTTP 200 with real book/work resource ids — the
DEFAULT_API_KEY from [1.1] has not rotated again yet.
Two full, real live runs on the Apify platform (testt0 account, actor
PyGpADimS6A9fJiQb), each against a different book, both finished
SUCCEEDED with real, non-empty review rows (rating, text, reviewer,
likes/comments) on the plain direct connection tier (no proxy escalation
needed) — see "Verified live" below for run ids and samples.
So the GraphQL/key-rotation self-healing itself is intact today. But the task
was to find why this Actor "keeps needing repeated maintenance" despite that
fix, so this session went further and exercised failure paths the [1.1]
session hadn't: a book id list mixing a working book with one Goodreads can't
resolve.
Root cause found this session: one bad/unresolvable book could take down the entire run and its already-collected data
Evidence. Running the Actor live with two book URLs — one a nonexistent
(but otherwise well-formed) book id, one a real book — produced a FAILED
run with zero dataset rows, even though the second, real book would have
returned reviews fine on its own (confirmed separately). The traceback showed
the first book's failure propagating all the way out of main() and
crashing the whole Actor:
RuntimeError: Could notopen the book page after 3 tries.
'message':"Variable 'legacyId' has an invalid value."}]
Two compounding bugs caused this:
src/main.py's book loop had no per-book error handling. Every URL in
urls was processed by one unguarded await fetch_reviews_for_book(...)
call inside the for loop in main(). Any book that ultimately failed
(wrong/rotated id, Goodreads returning an error for that specific book,
etc.) raised straight out of the loop, aborting every subsequent book in
the list and ending the run FAILED with whatever had (or, if it was the
first book, hadn't yet) been pushed. For any multi-URL input, a single
bad entry was enough to zero out the whole run — a very plausible
explanation for intermittent "keeps failing to produce output" reports
whenever a user's list included one stale/mistyped id.
src/helper.py classified every GraphQL error the same way — as a
possible-bad-apiKey "block" (GoodreadsBlocked), worth a key refresh, 3
retries, and two more full retry rounds through datacenter and
residential proxy escalation. Confirmed live against the real AppSync
endpoint that this conflates two very different, and very differently
actionable, situations:
An out-of-range/malformed legacyId → HTTP 200 with
"message": "Variable 'legacyId' has an invalid value." (no
errorType).
A well-formed but nonexistent book id (e.g. 2100000000) → HTTP 200
with "errorType":"RESOURCE_NOT_FOUND",
"message":"Book not found (maybe deleted, and no replacement exists).".
Neither is a block or a rejected key — Goodreads answered cleanly and
definitively both times. But the old code retried each one 3× per proxy
tier × 3 tiers before finally giving up, live-timed at ~68 seconds to
fail a single already-known-unresolvable book (see the 2n5Nbok5fEVaAPD2Q
run below) — wasting most of a short run's timeout/runtime on a foregone
conclusion, and, combined with bug 1, taking any other books in the list
down with it.
Changed
src/helper.py — added GoodreadsBookNotFound(RuntimeError), a new,
distinct exception for a confirmed, well-formed "no such book" GraphQL
response (any error without an auth-related marker in its errorType or
message — unauthorized, api key/apikey, forbidden,
accessdenied). _resolve_resource_ids_via_graphql() now only raises the
retry/escalate-worthy GoodreadsBlocked when an error actually looks
auth-related (or when the response has neither a book nor any error at all,
which still smells like a silently-rejected key); every other clean error
shape raises GoodreadsBookNotFound instead.
src/main.py — discover_api_config_with_retry() now re-raises
GoodreadsBookNotFound immediately instead of retrying it 3×;
_should_escalate() now returns False for it instead of triggering
proxy-tier escalation. A confirmed "no such book" answer fails in well
under a second instead of ~68 seconds.
src/main.py — main()'s per-book loop now wraps each
fetch_reviews_for_book() call in try/except: a failing book is logged
with a clear ❌ Skipping book i/n (...) warning and the run moves on to
the next URL instead of aborting. All per-book failures are summarized in
one warning block at the end. The run only raises (and fails) if zero
rows were collected across all requested books — a partial list of
good books now still produces a SUCCEEDED run with their real data, while
an all-bad list still fails clearly and fast instead of silently returning
an empty, green dataset.
Verified live
GET .../book/show/26032825 → HTTP 200; extracted
apiKey=da2-d2fyuybwsbf3poyquvbp2mbiwu; POSTed to AppSync getBookByLegacyId
→ HTTP 200, real ids for The Cruel Prince.
getBookByLegacyId for a valid-range but nonexistent id (2100000000) →
HTTP 200, errorType:"RESOURCE_NOT_FOUND" (confirms the new
GoodreadsBookNotFound classification is correct, not a block).
Before the fix — run 2n5Nbok5fEVaAPD2Q
(urls: 999999999999, 1885) → FAILED, 0 dataset rows, ~68s spent
retrying/escalating (direct → datacenter → residential) a book id that was
never going to resolve, before crashing before the second (valid) book was
ever attempted.
After the fix — run 3ljZGXNB7RbaG5KGB
(urls: 2100000000 [bad], 1885 [good]) → SUCCEEDED in ~3s total; bad
book skipped in ~0.1s with a clear ❌/⚠️ log, good book still saved 3
real reviews to the dataset (eMRtevpnGGTceh3eL, itemCount 3).
After the fix, all-bad input — run hHMeTqm45SzwuvaAs
(urls: 2100000000) → FAILED in <1s with a clear
RuntimeError: No reviews were collected for any of the 1 requested book(s)...
message instead of a slow, confusing proxy-escalation crash.
Two independent clean-input regression runs on the fixed build, each
SUCCEEDED with real data:
aEZzBZ3ssUGesRjQe — The Cruel Prince (id 26032825), 5/5 rows saved
(dataset T5GfjVbwQNCLgLpAE), e.g. reviewer "Melanie (meltotheany)",
5★, likeCount 3335.
03OJLNh8i0UV0WDde — The Hobbit (id 5907), sorted newest, 5/5 rows
saved (dataset qCzgNvwkif7gtL5pO), e.g. reviewer "Michelle", 5★, text
"my precious".
Confirmed this Apify account's plan (testt0, CUSTOM/BRONZE) does have
working RESIDENTIAL proxy group access (tested a direct proxied request
through groups-RESIDENTIAL — got a real residential exit IP), despite the
account API listing availableCount: 0 for that group — so
ProxyState's residential-escalation tier is not a dead end on this
account. Residential bandwidth is capped at 10 GB/month on this plan; if a
future run pattern relies on residential escalation heavily, that quota is
worth monitoring.
[1.1] - 2026-08-30
Fixed — broken run: Goodreads GraphQL API key rotation not being recovered from
Root cause. The Actor authenticates to Goodreads' public GraphQL (AWS AppSync)
API with a hardcoded x-api-key. Goodreads has since rotated that key —
da2-xpgsdydkbregjhpr6ejzqdhuwy (the old hardcoded value) now returns
HTTP 401 UnauthorizedException from AppSync, confirmed live against
https://kxbwmqov6jgg3daaamb744ycu4.appsync-api.us-east-1.amazonaws.com/graphql.
The Actor already had a designed fallback for exactly this ("refresh the key from
the page") but that fallback was itself broken in two independent ways, so every
run fell straight through to a hard failure instead of self-healing:
src/main.py built the fallback's target URL with to_reviews_page_url(),
i.e. the /book/show/<id>/reviews page. That page is served behind Goodreads'
AWS WAF JavaScript challenge and returns HTTP 202 (confirmed live) — so the
fallback always tried to scrape a page it could never actually load.
Even pointed at the correct, non-challenged /book/show/<id> page (no
/reviews suffix — confirmed live to return HTTP 200 with full HTML),
extract_api_data() in src/helper.py only looked inside external .js
bundle files for a "Production":{"apiKey":"...","endpoint":"https://..."}
pattern. Goodreads no longer ships the AppSync config that way at all: none
of the 26 JS chunks referenced from a live book page contain the strings
apiKey, appsync, or da2- any more. The key is now inlined directly in
the book page's own server-rendered HTML (Next.js SSR props), as
"dataSource":"Production","apiKey":"da2-...", with no adjacent
"endpoint" field at all — confirmed live (zero occurrences of the string
endpoint anywhere in the page). The old regex, which required an
"endpoint" key inside the same JS blob, could never match this shape.
Changed
src/helper.py — discover_goodreads_api_config() now always rebuilds the
fallback's target as the plain f"{BASE_URL}/book/show/{legacy_id}" URL from
the parsed numeric book id, instead of trusting whatever URL shape the caller
passed in. This guarantees the fallback never targets the WAF-challenged
/reviews page again, regardless of how it's called.
src/helper.py — _refresh_api_config_from_page() now runs extract_api_data()
against the fetched book page's own HTML first (that's where the key actually
lives now), before falling back to fetching and scanning its JS bundles. This
also cuts ~26 unnecessary HTTP requests per key refresh in the common case.
src/helper.py — extract_api_data() gained a second match path: a bare
"apiKey":"da2-..." pattern with no required adjacent "endpoint", paired
with the still-stable DEFAULT_ENDPOINT constant (the AppSync endpoint host
itself has not changed — only the key format/location did). The original
endpoint-paired regex is kept first, so a future reversion to the old bundle
format would still be picked up.
src/helper.py — DEFAULT_API_KEY updated to the currently valid key
observed live (da2-d2fyuybwsbf3poyquvbp2mbiwu), confirmed working end-to-end
against getBookByLegacyId for a real book id. This is a best-effort fast
path only; the fallback above is what keeps the Actor working through the
next rotation without a code change.
Verified live
Old hardcoded key against the real AppSync endpoint →
HTTP401 UnauthorizedException
(confirmed the break).
GET https://www.goodreads.com/book/show/<id>/reviews → HTTP 202, empty
body (confirmed the WAF challenge on the reviews page).
GET https://www.goodreads.com/book/show/<id> (no /reviews) → HTTP 200,
full HTML containing the live apiKey inline.
New key extracted from that HTML, POSTed to the AppSync endpoint with
getBookByLegacyId → HTTP 200 with real book/work resource ids
(The Cruel Prince, book id 26032825).
Full Actor run against the same book on Apify's infrastructure returned real,
non-empty review rows (rating, text, reviewer, likes/comments) — see the run
verification step in the session notes for the exact dataset sample.
