# Changelog of Goodreads Review Scraper (`fastdataapi/goodreads-review-scraper`) Actor

- **URL**: https://apify.com/fastdataapi/goodreads-review-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/fastdataapi/goodreads-review-scraper.md

## Changelog

All notable changes to this Actor are documented here.

### \[2.0] - 2026-10-18

#### Rebuilt against the current reference scraper (`Code/43 - Goodreads Review Scraper`)

The reference scraper this Actor was originally built from has since been rewritten
(confirmed live, 2026-10-15): Goodreads removed the static production AppSync
`apiKey` from its public JS bundle, so the old `DEFAULT_API_KEY` +
`getBookByLegacyId` discovery path this Actor relied on (added in \[1.1]) is no
longer the live mechanism Goodreads uses, even though the specific key value
happened to still authenticate at the time of this rebuild. The reference now
reads the live key and both work/book resource ids directly out of each
book's own `/reviews` page on every run — no hardcoded key, no separate
resource-id lookup call, and no stale-key maintenance going forward. This
session ported that discovery pipeline, verified end-to-end against the live
site (resource id extraction, live key extraction, review-service discovery,
and a real paginated review fetch all confirmed working), and re-platformed
its retry/pause/connection-escalation architecture onto Apify's own proxy
tiers instead of a single external one.

#### Changed

- `src/network.py` (new, replaces `src/helper.py`) — live discovery per book
  (no hardcoded or cached key), quick-retry + pause-and-retry rounds, and
  connection escalation via `Actor.create_proxy_configuration` (direct →
  user-configured Apify proxy → residential), replacing the previous
  aiohttp + hardcoded-key-then-page-scrape-fallback design.
- `src/parser.py` (new, replaces `src/review_extractor.py`) — pure parsing
  ported from the current reference, with two data-loss bugs fixed: the
  reference's own GraphQL query fetches `preReleaseBookSource` and five
  extra `shelf` fields (`displayName`, `editable`, `default`, `actionType`,
  `sortOrder`) but drops all of them when building its own output row
  (confirmed against its own `output.json` sample and against a live
  request in this session) — all six are now included. `viewerHasLiked`
  and `creator.viewerRelationshipStatus` are also carried through (always
  `null` for this Actor's logged-out requests, which is Goodreads' real
  answer, not a placeholder). The two `*Iso` timestamp fields added in an
  earlier version of this Actor are kept as an additive bonus beyond the
  reference's own output shape.
- `src/free_limits.py` (new) — free-plan usage limits (20 reviews per book
  per run, 200 per month across all runs), counted in a named key-value
  store (`free-quota`) in the running user's own account. Unlimited for
  local runs, Console test runs, and paying users.
- `src/main.py` — input field names now match the reference exactly:
  `urls` → `bookUrls`, the nested `filtersAndOptions.{sortBy,languageCode,
  reviewEdition}` → top-level `sortBy`, `languageCode`, `Review_Edition`.
  Bare numeric book ids are still accepted and rewritten automatically
  (kept from the previous build; the reference itself does not support
  this). Books are now processed with up to 4 running concurrently
  (previously strictly sequential). Every dataset push now follows the
  mandatory `charged_event_name="row_result"` + `event_charge_limit_reached`
  pattern exactly; the previous build's "disable charging forever after
  the first error" fallback is removed as unnecessary — pushes already
  succeed and save data even when PPE charging isn't configured for a run.
  `maxItems=0` (or negative) now fails the run with a clear validation
  error instead of silently falling back to a default.
- `.actor/input_schema.json`, `.actor/dataset_schema.json`,
  `.actor/output_schema.json` — updated for the new field names and the
  restored output fields; `proxyConfiguration` now actually feeds
  `Actor.create_proxy_configuration(actor_proxy_input=...)` (previously
  present in the form but not wired into the proxy logic at all).
- `Dockerfile` — added `ENV PYTHONUNBUFFERED=1` so log lines reach the
  Apify Console reliably during a real run.
- `requirements.txt` — added `curl_cffi` (browser TLS/HTTP impersonation,
  replacing the removed aiohttp-only HTTP layer) and `apify-client`;
  raised the `apify` floor to `>=2.3.0` for `charged_event_name` support.

#### Removed

- The reference scraper's hardcoded third-party residential-proxy
  credential (vendor: Evomi) was never ported — proxy fallback uses only
  Apify's own proxy configuration, as required.

#### Verified live (2026-10-18)

- Full discovery pipeline against a real book page: resource id extraction,
  live-key extraction, and review-service discovery from the page's own
  supporting resources all returned real, non-empty values.
- A real paginated review fetch (3 and 10-item requests) returned real
  review rows with every documented field populated, including the two
  previously-dropped `shelf` fields confirmed non-null in a live response
  (`displayName`, `editable`, `default`, `actionType`, `sortOrder`).
- A full local run (`bookUrls: ["26032825"]`, `maxItems: 3`,
  `sortBy: "newest"`, `languageCode: "en"`, `Review_Edition:
  "only_this_book"`) completed successfully end-to-end and saved 3 real
  review rows to the dataset.

### \[1.2] - 2026-09-07

#### Verified: the \[1.1] key-rotation fix is still working live

Before changing anything, this session re-verified the whole pipeline against
the real, current Goodreads site and AWS AppSync endpoint, live, rather than
trusting the previous fix:

- `GET https://www.goodreads.com/book/show/<id>` → `HTTP 200`, and it (and,
  today, even the `/reviews`-suffixed page — the WAF challenge from \[1.1] is
  currently not being served to this Actor's outbound IPs) contains the
  current inline `"apiKey":"da2-d2fyuybwsbf3poyquvbp2mbiwu"`.
- That exact key, POSTed to the AppSync endpoint for `getBookByLegacyId`,
  still returns `HTTP 200` with real book/work resource ids — the
  `DEFAULT_API_KEY` from \[1.1] has not rotated again yet.
- Two full, real live runs on the Apify platform (`testt0` account, actor
  `PyGpADimS6A9fJiQb`), each against a different book, both finished
  `SUCCEEDED` with real, non-empty review rows (rating, text, reviewer,
  likes/comments) on the plain `direct` connection tier (no proxy escalation
  needed) — see "Verified live" below for run ids and samples.

So the GraphQL/key-rotation self-healing itself is intact today. But the task
was to find why this Actor "keeps needing repeated maintenance" despite that
fix, so this session went further and exercised failure paths the \[1.1]
session hadn't: a book id list mixing a working book with one Goodreads can't
resolve.

#### Root cause found this session: one bad/unresolvable book could take down the entire run and its already-collected data

**Evidence.** Running the Actor live with two book URLs — one a nonexistent
(but otherwise well-formed) book id, one a real book — produced a `FAILED`
run with **zero** dataset rows, even though the second, real book would have
returned reviews fine on its own (confirmed separately). The traceback showed
the *first* book's failure propagating all the way out of `main()` and
crashing the whole Actor:

```
RuntimeError: Could not open the book page after 3 tries.
  ...
src.helper.GoodreadsBlocked: Goodreads GraphQL error: [{'errorType': None,
  'message': "Variable 'legacyId' has an invalid value."}]
```

Two compounding bugs caused this:

1. **`src/main.py`'s book loop had no per-book error handling.** Every URL in
   `urls` was processed by one unguarded `await fetch_reviews_for_book(...)`
   call inside the `for` loop in `main()`. Any book that ultimately failed
   (wrong/rotated id, Goodreads returning an error for that specific book,
   etc.) raised straight out of the loop, aborting every *subsequent* book in
   the list and ending the run `FAILED` with whatever had (or, if it was the
   *first* book, hadn't yet) been pushed. For any multi-URL input, a single
   bad entry was enough to zero out the whole run — a very plausible
   explanation for intermittent "keeps failing to produce output" reports
   whenever a user's list included one stale/mistyped id.
2. **`src/helper.py` classified *every* GraphQL error the same way** — as a
   possible-bad-apiKey "block" (`GoodreadsBlocked`), worth a key refresh, 3
   retries, *and* two more full retry rounds through datacenter and
   residential proxy escalation. Confirmed live against the real AppSync
   endpoint that this conflates two very different, and very differently
   actionable, situations:

   - An out-of-range/malformed `legacyId` → `HTTP 200` with
     `"message": "Variable 'legacyId' has an invalid value."` (no
     `errorType`).
   - A well-formed but nonexistent book id (e.g. `2100000000`) → `HTTP 200`
     with `"errorType":"RESOURCE_NOT_FOUND"`,
     `"message":"Book not found (maybe deleted, and no replacement exists)."`.

   Neither is a block or a rejected key — Goodreads answered cleanly and
   definitively both times. But the old code retried each one 3× per proxy
   tier × 3 tiers before finally giving up, live-timed at **~68 seconds** to
   fail a single already-known-unresolvable book (see the `2n5Nbok5fEVaAPD2Q`
   run below) — wasting most of a short run's timeout/runtime on a foregone
   conclusion, and, combined with bug 1, taking any other books in the list
   down with it.

#### Changed

- `src/helper.py` — added `GoodreadsBookNotFound(RuntimeError)`, a new,
  distinct exception for a *confirmed, well-formed* "no such book" GraphQL
  response (any error without an auth-related marker in its `errorType` or
  `message` — `unauthorized`, `api key`/`apikey`, `forbidden`,
  `accessdenied`). `_resolve_resource_ids_via_graphql()` now only raises the
  retry/escalate-worthy `GoodreadsBlocked` when an error actually looks
  auth-related (or when the response has neither a book nor any error at all,
  which still smells like a silently-rejected key); every other clean error
  shape raises `GoodreadsBookNotFound` instead.
- `src/main.py` — `discover_api_config_with_retry()` now re-raises
  `GoodreadsBookNotFound` immediately instead of retrying it 3×;
  `_should_escalate()` now returns `False` for it instead of triggering
  proxy-tier escalation. A confirmed "no such book" answer fails in well
  under a second instead of ~68 seconds.
- `src/main.py` — `main()`'s per-book loop now wraps each
  `fetch_reviews_for_book()` call in `try/except`: a failing book is logged
  with a clear `❌ Skipping book i/n (...)` warning and the run moves on to
  the next URL instead of aborting. All per-book failures are summarized in
  one warning block at the end. The run only raises (and fails) if **zero**
  rows were collected across **all** requested books — a partial list of
  good books now still produces a `SUCCEEDED` run with their real data, while
  an all-bad list still fails clearly and fast instead of silently returning
  an empty, green dataset.

#### Verified live

- `GET .../book/show/26032825` → `HTTP 200`; extracted
  `apiKey=da2-d2fyuybwsbf3poyquvbp2mbiwu`; POSTed to AppSync `getBookByLegacyId`
  → `HTTP 200`, real ids for *The Cruel Prince*.
- `getBookByLegacyId` for a valid-range but nonexistent id (`2100000000`) →
  `HTTP 200`, `errorType:"RESOURCE_NOT_FOUND"` (confirms the new
  `GoodreadsBookNotFound` classification is correct, not a block).
- **Before the fix** — run `2n5Nbok5fEVaAPD2Q`
  (urls: `999999999999`, `1885`) → `FAILED`, 0 dataset rows, ~68s spent
  retrying/escalating (direct → datacenter → residential) a book id that was
  never going to resolve, before crashing before the second (valid) book was
  ever attempted.
- **After the fix** — run `3ljZGXNB7RbaG5KGB`
  (urls: `2100000000` \[bad], `1885` \[good]) → `SUCCEEDED` in ~3s total; bad
  book skipped in ~0.1s with a clear `❌`/`⚠️` log, good book still saved 3
  real reviews to the dataset (`eMRtevpnGGTceh3eL`, itemCount 3).
- **After the fix, all-bad input** — run `hHMeTqm45SzwuvaAs`
  (urls: `2100000000`) → `FAILED` in <1s with a clear
  `RuntimeError: No reviews were collected for any of the 1 requested
  book(s)...` message instead of a slow, confusing proxy-escalation crash.
- Two independent clean-input regression runs on the fixed build, each
  `SUCCEEDED` with real data:
  - `aEZzBZ3ssUGesRjQe` — *The Cruel Prince* (id `26032825`), 5/5 rows saved
    (dataset `T5GfjVbwQNCLgLpAE`), e.g. reviewer "Melanie (meltotheany)",
    5★, likeCount 3335.
  - `03OJLNh8i0UV0WDde` — *The Hobbit* (id `5907`), sorted newest, 5/5 rows
    saved (dataset `qCzgNvwkif7gtL5pO`), e.g. reviewer "Michelle", 5★, text
    "my precious".
- Confirmed this Apify account's plan (`testt0`, `CUSTOM`/`BRONZE`) does have
  working `RESIDENTIAL` proxy group access (tested a direct proxied request
  through `groups-RESIDENTIAL` — got a real residential exit IP), despite the
  account API listing `availableCount: 0` for that group — so
  `ProxyState`'s residential-escalation tier is not a dead end on this
  account. Residential bandwidth is capped at 10 GB/month on this plan; if a
  future run pattern relies on residential escalation heavily, that quota is
  worth monitoring.

### \[1.1] - 2026-08-30

#### Fixed — broken run: Goodreads GraphQL API key rotation not being recovered from

**Root cause.** The Actor authenticates to Goodreads' public GraphQL (AWS AppSync)
API with a hardcoded `x-api-key`. Goodreads has since rotated that key —
`da2-xpgsdydkbregjhpr6ejzqdhuwy` (the old hardcoded value) now returns
`HTTP 401 UnauthorizedException` from AppSync, confirmed live against
`https://kxbwmqov6jgg3daaamb744ycu4.appsync-api.us-east-1.amazonaws.com/graphql`.
The Actor already had a designed fallback for exactly this ("refresh the key from
the page") but that fallback was itself broken in two independent ways, so every
run fell straight through to a hard failure instead of self-healing:

1. `src/main.py` built the fallback's target URL with `to_reviews_page_url()`,
   i.e. the `/book/show/<id>/reviews` page. That page is served behind Goodreads'
   AWS WAF JavaScript challenge and returns `HTTP 202` (confirmed live) — so the
   fallback always tried to scrape a page it could never actually load.
2. Even pointed at the correct, non-challenged `/book/show/<id>` page (no
   `/reviews` suffix — confirmed live to return `HTTP 200` with full HTML),
   `extract_api_data()` in `src/helper.py` only looked inside external `.js`
   bundle files for a `"Production":{"apiKey":"...","endpoint":"https://..."}`
   pattern. Goodreads no longer ships the AppSync config that way at all: none
   of the 26 JS chunks referenced from a live book page contain the strings
   `apiKey`, `appsync`, or `da2-` any more. The key is now inlined directly in
   the book page's own server-rendered HTML (Next.js SSR props), as
   `"dataSource":"Production","apiKey":"da2-..."`, with **no adjacent
   `"endpoint"` field at all** — confirmed live (zero occurrences of the string
   `endpoint` anywhere in the page). The old regex, which required an
   `"endpoint"` key inside the same JS blob, could never match this shape.

#### Changed

- `src/helper.py` — `discover_goodreads_api_config()` now always rebuilds the
  fallback's target as the plain `f"{BASE_URL}/book/show/{legacy_id}"` URL from
  the parsed numeric book id, instead of trusting whatever URL shape the caller
  passed in. This guarantees the fallback never targets the WAF-challenged
  `/reviews` page again, regardless of how it's called.
- `src/helper.py` — `_refresh_api_config_from_page()` now runs `extract_api_data()`
  against the fetched book page's own HTML first (that's where the key actually
  lives now), before falling back to fetching and scanning its JS bundles. This
  also cuts ~26 unnecessary HTTP requests per key refresh in the common case.
- `src/helper.py` — `extract_api_data()` gained a second match path: a bare
  `"apiKey":"da2-..."` pattern with no required adjacent `"endpoint"`, paired
  with the still-stable `DEFAULT_ENDPOINT` constant (the AppSync endpoint host
  itself has not changed — only the key format/location did). The original
  endpoint-paired regex is kept first, so a future reversion to the old bundle
  format would still be picked up.
- `src/helper.py` — `DEFAULT_API_KEY` updated to the currently valid key
  observed live (`da2-d2fyuybwsbf3poyquvbp2mbiwu`), confirmed working end-to-end
  against `getBookByLegacyId` for a real book id. This is a best-effort fast
  path only; the fallback above is what keeps the Actor working through the
  next rotation without a code change.

#### Verified live

- Old hardcoded key against the real AppSync endpoint → `HTTP 401
  UnauthorizedException` (confirmed the break).
- `GET https://www.goodreads.com/book/show/<id>/reviews` → `HTTP 202`, empty
  body (confirmed the WAF challenge on the reviews page).
- `GET https://www.goodreads.com/book/show/<id>` (no `/reviews`) → `HTTP 200`,
  full HTML containing the live `apiKey` inline.
- New key extracted from that HTML, POSTed to the AppSync endpoint with
  `getBookByLegacyId` → `HTTP 200` with real book/work resource ids
  (`The Cruel Prince`, book id `26032825`).
- Full Actor run against the same book on Apify's infrastructure returned real,
  non-empty review rows (rating, text, reviewer, likes/comments) — see the run
  verification step in the session notes for the exact dataset sample.

### \[1.0] - unknown

Initial version.