# Changelog of Trustpilot Reviews Scraper — Company Reviews & Trust Scores (`mfapitools/trustpilot-reviews-scraper`) Actor

- **URL**: https://apify.com/mfapitools/trustpilot-reviews-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/mfapitools/trustpilot-reviews-scraper.md

## Changelog — trustpilot-reviews-scraper

### unreleased — 2026-10-14 (Claude Opus 5.5) — committed, NOT deployed (deploy awaits the owner's OK)

- **Stopped steering customers onto residential proxy, which we pay for.** On a pay-per-event actor the developer pays the run's proxy bandwidth. The README called `useResidentialProxy` "strongly recommended", its example input set it to `true`, the input description said datacenter IPs are blocked, and every datacenter run logged a WARNING telling the customer to switch — while the README's own note measured a 5-review residential run at ~$0.031 of cost against $0.02 of revenue. Datacenter + session retirement passes the WAF (Apify's daily QA runs that way: 26 of 27 public runs in 30 days succeeded). README, input description and the startup log now say: leave it off, turn it on only if runs fail with WAF blocks; the failure messages still suggest it, i.e. only after datacenter actually failed. No code path or price changed. 128/128 tests.
- **README price text corrected to the live prices (owner approved 2026-10-14).** It said $0.003/review, $0.0001/company profile and a $0.02 start fee, with cost examples built on those; live prices (`GET /v2/acts/GIsLrXf9NR4dCbyMh`, 2026-10-14) are $0.002, $0.0005 and $0.005 per GB ($0.01 at the default 2048 MB). Examples now ~$0.11 / ~$0.61 / ~$2.01 / ~$0.41; "$30 per 10,000" → "$20". Removed the internal "slightly loss-making for the developer" cost note from the store listing. Prices themselves unchanged.

### build 1.1.20 — 2026-07-30 (fixes "Under maintenance"; Claude Sonnet worker agent, verified by Opus 5)

- **Root cause: a WAF challenge page was being accepted as a successful scrape.** Trustpilot's AWS WAF returns HTTP 200 with a challenge body and no `#__NEXT_DATA__`. The old code logged and returned, so Crawlee counted the request as *finished* and never retried it (`retryHistogram: [1]` on the failed run). One blocked attempt killed the whole run. This — not the 5-minute limit — was the real driver of the FAILED runs, and of much of the timing variance too (a pre-fix run burned 180s retrying `page.goto` on the SAME exhausted session before Crawlee's own error-score threshold finally rotated it).
- **Fix:** a challenge page now throws a dedicated `WafChallengeError` from the request handler, and an `errorHandler` calls `session.retire()` before every retry so the next attempt gets a fresh proxy identity immediately instead of waiting three strikes. `maxRequestRetries: 5`, `navigationTimeoutSecs: 40`. If every attempt is blocked it still exits 1 loudly — a real block never becomes a silent zero-result success.
- **Prefill made QA-sized:** `companyDomains` → `["notion.so"]` (was `["amazon.com"]`, one of the largest review corpora on the site) and `maxReviewsPerCompany` prefill `20`. `default` stays 100, so paying customers are unaffected — a prefill is a demo default and a QA budget, not a cap.
- **Residential proxy deliberately NOT enabled by default.** It works (run `LecizWbH7gAaxYaoD`: 101 reviews, 168.8s) but its bandwidth cost would consume roughly the entire $0.045 revenue of a 20-review run. `useResidentialProxy` stays default false and remains the documented escape hatch.
- **Verified: 6/6 prefill-only runs green** — `eZVAJ6Pr026N9r2Uv` 49.1s, `Z3qyjkybdOZl4BeuX` 16.0s, `HFM2cqu1jOpUcIeVZ` 28.4s, `dsMsLJxkXK1khR1ZK` 71.1s, `UzU51WczVLWiakUPY` 38.5s, `G0OzlNGY9mC3oZdFY` 25.7s, 21 items each. Worst case 71s against QA's 300s limit. Before the fix the same input gave 29s / 31s / 215s / FAILED.
- Memory stays 2048MB via the top-level `defaultMemoryMbytes` key (the nested `defaultRunOptions.memoryMbytes` is silently ignored by Apify — do not remove the top-level key).

### build 1.1.17 — 2026-07-25

- **Live-validation fix**: `@money/shared`'s `buildProxyConfig()` requested a proxy group literally named `'DATACENTER'`, which does not exist on this account (real group: `BUYPROXIES94952`) — every run with `useResidentialProxy: false` (the default) silently fell back to no proxy at all, with a caught `Invalid username` error logged on every run. Fixed the group name in the shared lib. Behavior change: default (non-residential) runs now actually route through the shared datacenter proxy pool instead of a direct connection. Re-verified on the platform with the exact default input (`companyDomains: ["amazon.com"]`, `useResidentialProxy: false`): run `TfjxHPWQ8ombAgc9W` succeeded, 100 reviews + 1 company profile, no regression — unlike computrabajo, this actor's WAF-handling warning already tells customers to use residential for reliability, so this is a straightforward correctness fix with no observed downside here. | Claude Sonnet (live-validation agent)

### build 1.1.14 — 2026-07-24

- **Competitor-standard input filters** (closes the #1-ranked gap in `research/competitor-teardown-2026-07.md`: "Input-level filters (star rating, date-range presets, language, verifiedOnly, repliesOnly) — 3 of 4 competitors have all of these, we have none beyond `sinceDate`"). New inputs, all off by default (no behavior change unless set):
  - `starRatings` (integer\[] 1-5, `editor: select`) — only collect reviews with these ratings.
  - `languages` (string\[], `editor: stringList`) — only collect reviews in these language codes (case-insensitive match against a new `language` field, added to the `review` record output, sourced from Trustpilot's raw `review.language`).
  - `verifiedOnly` (boolean) — only collect reviews Trustpilot marks "verified".
  - `withRepliesOnly` (boolean) — only collect reviews with a public company reply.
  - **Server-side vs post-parse decision**: fetched `trustpilot.com/robots.txt` directly (HTTP 200, no browser/proxy needed) and confirmed `stars=`, `languages=`, `verified=`, `replies=` are real, functional Trustpilot query params — but every named crawler section explicitly disallows all four (`Disallow: /*?*stars=` etc.), consistent with `date=` (already disallowed, which is why `sinceDate` has always been a post-parse filter, not a `?date=` param). All four new filters follow that existing precedent: implemented as **client-side post-parse filtering only** — `buildReviewUrl` is untouched, still emits only `?page=N&sort=`. No new query params were added to the request surface.
  - Filters compose with AND, and compose correctly with `sinceDate`/`sortBy`/`maxReviewsPerCompany`: the `sinceDate` pagination-cutoff check (`hitDateCutoff`) is computed from date-filtering alone, *before* the new filters are applied, so a page where every review fails e.g. `starRatings` does not incorrectly look like "we've paginated past the requested date range" and stop pagination early.
  - Charging unchanged: `review-scraped` is only charged for reviews that pass every active filter and are actually pushed to the dataset (filtered-out reviews were never candidates for `Actor.pushData`/`chargeForItem` in the first place).
  - New pure functions in `src/parsers.ts`: `matchesStarRatings()`, `matchesLanguages()`, `matchesVerifiedOnly()`, `matchesRepliesOnly()`, `passesReviewFilters()`, `filterReviews()` — 29 new unit tests (128 total, all green), covering each filter individually, AND-composition across all four, empty-filter no-op behavior, the new `language` field mapping in `buildReview()`, and `buildReviewUrl()` never emitting `stars=`/`languages=`/`verified=`/`replies=`.
  - README: new "Filters" section (table + two worked examples + cost note on narrow filters paging through more results) and updated input/output tables; `.actor/input_schema.json` updated with the four new properties.
  - **Input schema fix mid-deploy**: first push (build 1.1.14) failed platform input-schema validation — `select`-editor array items must be `type: "string"` on Apify, not `"integer"`. Fixed by keeping `starRatings` as string enum `["1".."5"]` in `.actor/input_schema.json` and adding a zod `.transform().pipe()` step in `main.ts` that accepts both strings and numbers and coerces/validates to `number[1-5]`. Re-pushed as build 1.1.15 (succeeded).
  - **Platform validation** (build 1.1.15, run [`fu1wDJjElhIhtPoIB`](https://console.apify.com/actors/GIsLrXf9NR4dCbyMh/runs/fu1wDJjElhIhtPoIB)): `companyDomains: ["amazon.com"]`, `starRatings: ["1","2"]`, `languages: ["en"]`, `maxReviewsPerCompany: 5`, `useResidentialProxy: true`. Run **SUCCEEDED** — 1 company profile + 5 reviews pushed. Verified via dataset (`gup548D54oqaEx1uC`): all 5 returned reviews have `rating` 1 or 2 (ratings observed: 2,1,1,1,1) and `language: "en"` — filters confirmed working end-to-end on live platform infrastructure.

### build 1.1.13 — 2026-07-24

- **Orchestrator review follow-up on the 1.1.12 guards (two product-risk fixes):**
  - `maxRunSeconds` default changed **240 → 0 (disabled)**. A hidden 240s deadline would silently truncate legitimate large runs where the customer raised the platform timeout; the zero-yield guard plus the platform timeout already bound the pathological cost cases. The deadline is now strictly opt-in (scheduled runs with a hard time budget).
  - **Below-threshold silent-failure catch**: 1.1.12's own validation run (`HlhbKdtRqbFd8a5Jy`, single WAF-blocked domain) exited 0 with 0 items because one page can't reach the 3-page guard threshold — the exact silent-failure class AGENTS.md bans. New pure function `isSilentZeroYield()`: any run ending with zero yielded pages AND zero items (including "no page ever reached the handler") now exits 1 with a clear statusMessage. A legitimate 0-review company ('yielded' page) never trips it. 5 new tests (99 total, green).

### build 1.1.12 — 2026-07-24

- **Zero-yield guard (protective fix — bounds the measured "0 items, full compute cost" loss shape).** Real platform run `lBgcDGfoXRfl1D7p3`-class failures (66s Playwright crawl, $0.041 compute, 0 items) previously ran to completion and only got covered by the flat actor-start charge — thin margin, no fast-fail. Now:
  - Distinguishes three "no data" cases instead of conflating them: **(a) WAF/challenge block** (no `__NEXT_DATA__` in the DOM), **(b) parse failure** (`__NEXT_DATA__` present but malformed or missing `businessUnit` — structural drift), and **(c) company genuinely has 0 reviews** (parsed fine, real profile extracted, 0 reviews — a legitimate result, not a failure).
  - New input `zeroYieldAbortThreshold` (default `3`, `editor: number`): if the first N processed pages are ALL case (a)/(b) failures, the crawl aborts immediately, logs the reason, and exits `exitCode: 1` with a clear `statusMessage` — never triggered by case (c). Only looks at the start of the run (not a rolling window), so a late-crawl failure streak after a productive start never trips it.
  - New input `maxRunSeconds` (default `240`, `editor: number`): belt-and-suspenders soft deadline checked between pages. On breach, stops cleanly with whatever was collected — exit `0` (partial result) if any items were produced, exit `1` if none were.
  - New pure functions in `src/parsers.ts`: `classifyPageYield()`, `evaluateZeroYieldGuard()`, `isRunDeadlineExceeded()` — unit tested (21 new tests, 94 total, all green) without touching Playwright/network.
  - Charge semantics unchanged: no charging for items never pushed; `eventChargeLimitReached`/`budgetExhausted` behavior untouched, checked independently of the new guards.
  - README: new "Zero-yield guard (fast failure detection)" section under Limitations; input table updated.

### build 1.1.11 — 2026-07-24

- **README fix**: the MCP setup snippet pointed to a competitor's actor (`getwally.net/trustpilot-reviews-scraper`) instead of ours (`mfapitools/trustpilot-reviews-scraper`) — copy-paste bug from the listing revamp; would have routed AI agents to the competitor's tool. Fixed and pushed (README-only, no code change).

### v1.1.0 — 2026-07-23

- **Migrated from CheerioCrawler to PlaywrightCrawler** (headless Chromium). Plain HTTP fetchers are permanently blocked by Trustpilot's AWS WAF; Playwright renders the JS challenge and receives real page content.
- **Parser validation against real pages**: fetched real HTML from notion.so and asana.com via headless Playwright. Corrected all field names that differed from synthetic assumptions:
  - `review.rating` (was assumed `.stars`)
  - `review.dates.publishedDate` (was assumed `.createdAt`)
  - `review.isPending` (was assumed `.pending`)
  - `review.labels.verification.isVerified` (was assumed `.isVerified`)
  - `review.reply.message` (was assumed `.companyReply.text`)
  - `businessUnit.isClaimed` (was assumed `.claimed`)
  - `businessUnit.numberOfReviews` is a plain number (was assumed `{total: N}` object)
  - `businessUnit.categories[0].id` (was assumed `.categoryOrSubcategoryId`)
  - Pagination at `pageProps.filters.pagination` (was assumed `pageProps.pagination`)
- **Added `extractNextDataFromHtml()`** — Playwright variant for extracting `__NEXT_DATA__` from raw HTML without Cheerio.
- **Added `isChallengeTitle()`** — Playwright variant of WAF detection without Cheerio.
- **Added `getPagination()`** — extracts pagination from correct nested location (`filters.pagination`).
- **Unit tests updated**: 73 tests, all running against real notion.so fixtures (real `__NEXT_DATA__` from live page). Legacy synthetic fixture kept for WAF-detection tests only.
- **Dockerfile**: switched to `apify/actor-node-playwright-chrome:22` base image. Playwright listed in `package.docker.json` with `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1` (browser is preinstalled in base image).
- **Memory default**: increased to 2048 MB (`defaultRunOptions.memoryMbytes`) to accommodate Chrome.
- **WAF handling**: removed 403 from `sessionPoolOptions.blockedStatusCodes` (Trustpilot always returns 403 HTTP even for valid pages rendered by Playwright).
- **Bandwidth optimization**: blocks image/font/media/stylesheet routes via Playwright route interception.
- **esbuild**: added `--external:playwright --external:playwright-core`.
- **DOM extraction**: switched from `page.content()` + regex to `page.evaluate(() => document.getElementById('__NEXT_DATA__').textContent)` for more reliable `__NEXT_DATA__` extraction (avoids intermediate HTML state races).
- Local e2e: 5 real notion.so reviews scraped successfully from local/residential IP.
- Platform runs: 2 builds deployed (1.1.1, 1.1.2 fixed `playwright` in `package.docker.json`). Build 1.1.4 current with DOM evaluate approach. Platform WAF validation pending owner-authorized run (budget used up during debugging).

### v0.1.0 — 2026-07-23

- Initial implementation.
- Parses `__NEXT_DATA__` JSON blob from Trustpilot Next.js SSR pages.
- Outputs `review` records and optional `company-profile` records.
- Supports `companyDomains`, `startUrls`, `maxReviewsPerCompany`, `sinceDate`, `sortBy`, `includeCompanyProfile`, `useResidentialProxy` inputs.
- Pagination up to page 20 (public cap documented).
- Parser unit tests offline against HTML fixture.
- Pay-per-event charging: `review-scraped` + `company-profile` events.
- Compliance: public data only, reviewer display names only, no PII.