# Changelog of SmartCustomer Review Scraper (Pay-Per-Event) (`prodiger/smartcustomer-review-scraper`) Actor

- **URL**: https://apify.com/prodiger/smartcustomer-review-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/prodiger/smartcustomer-review-scraper.md

## Changelog

### v0.4 — 2026-07-25 — Business responses on every review

Reviews now carry the **business's public reply** in a new `business_response` field — `{ text, published, author_name }`, or `null` when the business hasn't replied. This is what shoppers and support/ORM teams actually want alongside the review itself.

#### What changed

- **New field: `business_response`.** The business's reply text (`text`), when it was posted (`published`), and who posted it (`author_name`, usually the business). `null` when there is no reply.
- **New input flag `includeBusinessResponses`** (default `true`). Turn it off to omit responses. **No extra charge** — responses ride along with the reviews you already collect; billing is unchanged (`business-resolved` + `review-extracted`).
- **Review source switched from the HTML page's JSON-LD to SmartCustomer's public reviews API** (`api.smartcustomer.com/v2/businesses/<domain>/reviews`). The server-rendered JSON-LD does **not** expose business replies; the API returns each reply inline (`business_comment`). The API is also the same endpoint the SmartCustomer website itself reads, is ~20× lighter than the 900 KB+ HTML page (lower proxy cost per run), and provides structured, stable fields.
- **Richer business metadata for free:** `business.categories` is now populated from the business subindustry (previously always `[]`), and `business.review_count` reflects the business's total review count from the API's pagination metadata.

#### Compatibility

- **Output field shapes are unchanged** — `review_id`, `review_url`/`url`/`business_url`, `text`, `rating`, `review_date`, `user_name` (still "First L."), `user_avatar_url`, `photos[]`, `status`, embedded `business`, `scraped_at` are all preserved. `business_response` is purely additive.
- `sortBy` remains accepted but is a no-op — the SmartCustomer site and its API both ignore sort parameters and serve a fixed "recommended" order (this was already true of the HTML source in v0.3).
- Verified: `tsc` build clean, 94 unit tests pass (incl. a new `reviews-api` suite), plus a live end-to-end smoke test (fetch → parse → record) confirming real business responses attach correctly (Shein 37/40, Fashion Nova 2/40, Chewy 0/40) with full field integrity, and unknown domains resolve to zero rows.

### v0.3 — 2026-05-07 — Canonical SmartCustomer rebrand + competitor-matched PPE

The site previously branded as **Sitejabber** has rebranded to **SmartCustomer** — `sitejabber.com` 301-redirects to `www.smartcustomer.com`, and the rendered page's `<title>` reads "SmartCustomer". v0.2 documented this in passing but kept "Sitejabber" in the actor title and most user-facing copy. v0.3 normalizes the canonical brand to SmartCustomer everywhere:

- Actor title, description, and Apify Store metadata → "SmartCustomer Review Scraper".
- README, CHANGELOG, input-schema descriptions, code comments → SmartCustomer.
- `buildReviewsUrl` now hits `https://www.smartcustomer.com/reviews/<domain>` directly (skip the sitejabber.com 301 hop).
- Permalinks emitted on review records use the smartcustomer.com host (already true via the v0.2 host allowlist; explicit now).
- The legacy sitejabber.com host stays in the read-side allowlist so existing buyer inputs that reference the old domain still canonicalize cleanly.

#### Pricing — matched to working SmartCustomer competitors

Surveyed Apify Store competitor PPE pricing for the SmartCustomer/Sitejabber target:

- `cerridwen/sitejabber-scraper` (54 runs): `apify-actor-start` $0.00005 + **$0.0015 per result**.
- `zadexinho/sitejabber-scraper` (99 runs): `apify-actor-start` $0.00005 + per-review event.

v0.2 priced `business-resolved` $0.001 + `review-extracted` $0.0005 — roughly 3× cheaper than `cerridwen` per result, leaving margin on the table. v0.3 repriced to match the working market:

| Event | v0.2 | v0.3 |
|---|---|---|
| `apify-actor-start` (auto-fired on every run) | not configured | $0.00005 |
| `business-resolved` (one per resolved business) | $0.001 | $0.0005 |
| `review-extracted` (one per emitted review, post-`onlyWithPhotos` filter) | $0.0005 | $0.0015 |

A 50-review run from one business will cost `$0.00005 + $0.0005 + 50 * $0.0015 = $0.07555` — competitive with `cerridwen`'s `50 * $0.0015 = $0.075`.

**Pricing change is gated by Apify's 1-month cooldown between price modifications.** v0.2 pricing was set on 2026-05-07; the next pricing change is allowed on 2026-06-07. The v0.3 prices above will be applied then, with another 14-day notification period after that. Until 2026-06-07 the actor runs on the v0.2 pricing ($0.001 + $0.0005, effective 2026-05-22).

### v0.2 — 2026-05-05 — Pivot from Yelp to SmartCustomer

**Breaking change.** v0.1 targeted Yelp; v0.2 targets [SmartCustomer](https://www.smartcustomer.com).

#### Why the pivot?

Yelp's WAF (DataDome) comprehensively blocks every IP in Apify's RESIDENTIAL, BUYPROXIES94952, and StaticUS3 proxy pools. Six iterations of stealth Playwright, cookie warming, and headful-Chrome fingerprint patches did not get us past the challenge. Even Apify's flagship `super-scraper-api` (a drop-in scraper-API replacement for ScrapingBee) returns "Request blocked" against Yelp on Apify proxies. Wrapping working competitors (`tri_angle/yelp-review-scraper`) is plan-blocked on the user's CUSTOM Bronze plan.

**SmartCustomer** uses much softer anti-bot defenses (HTTP 200 from Apify RESIDENTIAL on first request) and server-renders 20 reviews per page in a JSON-LD `Organization` block with a stable `aggregateRating` + `review[]` schema. Same architecture (PPE, dual-input, embedded business metadata, photos validation, run summary) ports cleanly.

#### What's the same

- **Pay-per-event pricing**: `business-resolved` (one per resolved business) + `review-extracted` (one per emitted review, post-`onlyWithPhotos` filter).
- **Output schema**: every review record carries `review_url` / `url` / `business_url` (first-present-wins permalink), `text`, `rating`, `review_date`, `user_name`, `user_avatar_url`, `photos: string[]`, `status`, embedded `business` block, `scraped_at`.
- **Run-level summary** record at end of run with aggregate counts.
- **Per-record `status` enum**: `ok` / `partial` / `business_not_found` / `blocked` / `rate_limited`.
- **SSRF defense**: strict host allowlist on every URL the actor navigates to or stores. Photo URLs validated against `*.smartcustomer.com` (SmartCustomer's image CDN).
- **`onlyWithPhotos` flag** preserved.

#### What changed

- **Input shape**: `domains: string[]` (e.g., `["amazon.com"]`) replaces Yelp's `searchTerms` + `businessAliases`. `businessUrls` now accepts SmartCustomer `/reviews/<domain>` URLs.
- **Anti-bot stack**: removed Playwright/stealth/cookie-warming code. Reverted to Crawlee `CheerioCrawler` + got-scraping TLS impersonation. Dockerfile back to `apify/actor-node:24` (Alpine).
- **Pagination**: SmartCustomer paginates via `?page=N` (20 reviews per page).
- **Memory**: `defaultRunOptions.memoryMbytes` reverted to 1024.
- **Search-term resolution** is reserved for v0.3 — SmartCustomer's search surface doesn't map cleanly to the two-stage flow yet.

#### Known limitations (SmartCustomer-specific)

- **Photo coverage is inconsistent.** SmartCustomer surfaces user-uploaded review photos for some businesses but not others. Many reviews emit `photos: []`. The `onlyWithPhotos: true` filter still works correctly but will drop most reviews on photo-light businesses.
- **Per-page review count varies.** Most pages return 20 reviews server-rendered; some return fewer when SmartCustomer serves a partially client-hydrated variant.
- **No `categories` or `price_tier`** in business metadata (SmartCustomer doesn't expose these). Both fields emit as `null` / `[]` for schema compatibility with the v0.1 contract.

#### Yelp-specific empirical findings (preserved as institutional learning)

| Attempt | Result |
|---|---|
| CheerioCrawler + RESIDENTIAL | 403 (DataDome challenge HTML) |
| PlaywrightCrawler + RESIDENTIAL (default fingerprints) | Crawlee aborts on 403 |
| PlaywrightCrawler + manual goto + 25s wait + headful xvfb | DataDome markers persist after 41s |
| PlaywrightCrawler + stealth init + cookie-warming homepage | Both homepage AND target return 403 + DataDome challenge |
| Apify `super-scraper-api` standby endpoint with `premium_proxy=true` | 500: "Request blocked - received 403 status code" |
| Yelp `/biz/<alias>/review_feed` JSON endpoint via raw proxy | 403 + `geo.captcha-delivery.com` redirect URL |
| `m.yelp.com/biz/<alias>` mobile site | 403 + DataDome `'rt':'c'` (visual CAPTCHA challenge) |
| BUYPROXIES94952 proxy group (datacenter, 27 available) | 403 + DataDome challenge |
| StaticUS3 proxy group (3 static IPs) | 403 + DataDome challenge |
| DATACENTER proxy group | 407 (group not available on this plan) |

DataDome on Apify's standard proxy infrastructure in 2026 is unbeatable with code alone.

### v0.1 — Initial Yelp release (superseded)

Architecturally complete actor targeting Yelp with PPE, dual-input, embedded business metadata, Yelp-host-validated photos, and run-level summary record. Functionally non-functional in production due to the DataDome block above. See v0.2 entry for the pivot rationale.