# Changelog of Amazon Product Scraper – Prices, Ratings, Offers & Sellers (`badr_xba/amazon-product-scraper`) Actor

- **URL**: https://apify.com/badr_xba/amazon-product-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/badr_xba/amazon-product-scraper.md

## Changelog

### 2026-11-12 (graceful stop hardening)

- The stop before the timeout is a watchdog: it cuts a slow page, a retry pause or a browser visit too (before, a single long step could still run into the platform abort). Margin kept to save: max(45 s, 10 % of the timeout) (was 30 s). Retry pauses never sleep past the stop; closing the browser is bounded. Without `ACTOR_TIMEOUT_AT`/`APIFY_TIMEOUT_AT`: start + `ACTOR_TIMEOUT_SECS` when given, else (on Apify) start + the run's timeout from the run options (API), else the Actor default 3600 s. The final visual report is bounded by the time left (skipped under 15 s; FAILED_INPUTS and RUN_SUMMARY are saved before it). Output and billing unchanged.

### 2026-11-12 (zero failures)

- **Residential fallback, capped and measured.** A page Amazon keeps refusing (503, robot check, error page) is retried 5 times from new sessions on the customer's proxy, then up to 3 times through the RESIDENTIAL group — only when the customer uses the default Apify proxy (never when they chose RESIDENTIAL or their own proxies). Every fallback request's wire bytes are counted (curl's download + header + request sizes, +5 %); the fallback stops for the run at `maxFallbackRequests` (new input, default 100, 0 = off) or at the cost cap: $0.01 + $0.00144 per saved product at $8/GB, which keeps ≥ 30 % margin at the lowest product price ($0.003). Sites that keep blocking get only 2 cheap tries before the fallback. CAPTCHAs are never solved; nothing logs in.
- `RUN_SUMMARY`: `fallbackRequests`, `fallbackBytes`, `estimatedFallbackCostUsd`, `fallbackPagesRecovered`, `fallbackAllowanceUsd`, `fallbackStopReason`, `maxFallbackRequests`; stop reason `timeLimit`.
- **No more FAILED runs for customer-side problems**: a run where everything was refused (or nothing was valid) now ends SUCCEEDED with a clear status message and `FAILED_INPUTS` (was FAILED). New input `failRunOnError` (default false) restores a FAILED status for automations that need it (any failed input or nothing saved).
- Error code `blocked` is now `BLOCKED`; new code `TIME_LIMIT`.
- **Graceful stop before the timeout** (`ACTOR_TIMEOUT_AT` / `APIFY_TIMEOUT_AT`): no new page 30 s before, 10 s for pages in flight, then everything is saved and the run ends SUCCEEDED; unfinished start URLs go to `FAILED_INPUTS` as `TIME_LIMIT`. A "Still working" log line at most every 30 s.
- Input schema rejects malformed input before the run starts: `countryCode` (2 letters), `zipCode` (2–11 letters/digits/space/dash), `runTag` (≤ 100 characters), `maxFallbackRequests` (0–1000).
- `minMemoryMbytes: 512` in actor.json (production peak 225–338 MB, stats.memMaxBytes of the last runs).
- Billing unchanged: same events, same prices; the fallback is paid by the Actor, never billed to the customer.

### 2026-11-09 (review invitation)

- Successful runs that saved at least one result end their status message with one short, optional invitation to leave a Store review or open an issue (never on failed or empty runs); README section "Feedback and reviews". Output and billing unchanged.
- Fixed: a malformed link (`https://[…`) or a list sent as `language` / `amazonDomain` through the API crashed the whole run (FAILED) before any page was read; now reported in FAILED_INPUTS or ignored.
- Hardened: a failed progress-message update, a browser cleanup error or a failed FAILED_INPUTS/RUN_SUMMARY write no longer turns a run that saved products into FAILED (logged as a warning).

### 2026-11-08 (b) — visual kit 1.1.0

#### Added

- Visual report on kit 1.1.0: at most 3 photos per product (main photo first, card thumbnail strip), data-quality
  panel (price, rating, reviews, brand, availability, seller, category, rank, delivery), failed inputs grouped by
  error code, and a **live** report while the run is going (start URLs done/total, rewritten at most once a minute).
  The final report is also written when the run crashes (status FAILED) instead of staying "Running".
- Optional `runTag` input, copied to every row as `runTag` (absent when not set; nothing else changes).
- `errorCode` in each FAILED_INPUTS entry (same value as `error`); error-code table in the README.
- Dataset view *Ratings and best-seller ranks* (image first); output schema link to it.

#### Fixed

- `bestsellerRanks`: the "(See Top 100 in …)" link was parsed as an extra rank `{rank: 100, category: "… )"}`
  (4 of 10 rows of production run nqa8sC7bu3HF24DIO). Parenthesised text is now ignored.

### 2026-11-08

#### Added

- **Visual report** (`REPORT` record, Output tab → *Visual report*): product cards with photo gallery (300 px in the
  grid, 1,500 px in the viewer), price, list price, discount, rating, best-seller rank; compare mode (up to 4);
  key figures, price/discount/brand charts, search, filters, sorting, table view and CSV/JSON export of the filtered
  rows. Built by the shared `src/visual_kit.py` (v1.0.0) with this Actor's `src/report.py`. At most 2,000 rows are
  kept in memory for it; it is never billed and a report error is only logged as a warning.
- *Offers and sellers* view: product image column. Output schema: *Visual report* link; key-value store schema:
  `REPORT` (text/html).

#### Fixed

- Image links (`thumbnailImage`, `galleryThumbnails`, `highResolutionImages`) are normalized: `//host/…` becomes
  `https://host/…`; `data:` URIs and Amazon's lazy-load placeholder (`x-locale/common/grey-pixel.gif`) are dropped,
  and for the main image the next real source on the page is used. Valid links are unchanged.

#### Tests

- `tests/test_report.py`: 12 tests on 5 real products (image ownership, gallery pairing, placeholders, URL
  normalization on the saved pages, escaping, a 5,000-row synthetic set, export = embedded data, failure safety,
  rows pushed → report).

### 2026-11-07

#### Fixed

- Best-seller lists (and every list price read from an `.a-price` block): prices came out 100× too high when Amazon's
  hidden price text had no decimal separator (`MAD17477` was saved as 17477.0 instead of 174.77). The visible
  whole + fraction parts are now used first. Affects rows saved from the list (product details off, or when a product
  page could not be read).
- A search, category or best-seller URL answering 404 was retried 8 times from new proxy addresses (wasted proxy
  traffic) and then reported as `blocked`; it now ends at once as `no_results_found`.
- Values of the wrong type in the URL / keyword boxes (numbers, booleans, nested lists sent through the API) were
  silently dropped and, when nothing else was given, the example "keyboard" search ran instead. They are now listed in
  FAILED_INPUTS (`invalid_url`, not billed) and the example search runs only for a truly empty input.

#### Improved

- Retries use exponential backoff with jitter (~0.5, 1, 2, 4, 8 s) and no longer hold a concurrency slot while
  waiting, so one page being retried does not slow the others. Short links (amzn.to, a.co) are retried on network
  errors and 5xx answers.
- Product and list pages are parsed in a worker thread (~0.5 s of CPU per product page), so downloads continue
  while a page is parsed.
- Two start URLs listing the same product at the same time fetch its page once.
- Every saved row is checked against the dataset schema types; safe conversions are applied, anything else is
  logged once.
- Dataset schema: descriptions say which fields are often empty and why.
- `requirements.txt`: every dependency upper-bounded (`parsel<2`, `playwright<2`, compatible with the base image).
- Tests: `python -m pytest -q` (offline, real saved pages in `tests/fixtures`). `tests/` is excluded from the
  Docker image.
