# Changelog of Walmart Product, Price and Review Data Scraper (`khadinakbar/walmart-data-extractor`) Actor

- **URL**: https://apify.com/khadinakbar/walmart-data-extractor/changelog.md
- **Full Actor documentation**: https://apify.com/khadinakbar/walmart-data-extractor.md

### 0.9 — 2026-06-12

- SerpApi paid fallback for anti-bot recovery: when Walmart's Akamai/PerimeterX blocks every rotated US residential session and the live scrape yields 0 products, the same search/itemId/productUrl request is routed to SerpApi's managed Walmart engine (`SERPAPI_KEY` account secret) so the run returns REAL product data instead of an empty diagnostic row. Recovered products go through the same cap/charge/dedup path and the run exits SUCCEEDED. SerpApi search fields are shallower than a full scrape (no variants/specs/reviews), so the crawler stays primary and SerpApi fires only on block-exhaustion. Undocumented `serpApiOnly` input flag routes directly to the managed source.

### 0.8 — 2026-06-11

- Never-FAILED hardening: a run can now only end FAILED on a genuine unrecoverable code crash. (1) `crawler.run()` is wrapped in try/catch — a fatal Crawlee/proxy/browser-launch error now writes a schema-valid diagnostic row and exits SUCCEEDED with a terminal WARNING status instead of propagating uncaught. (2) New `safePushData` wraps EVERY `Actor.pushData` (products, reviews, diagnostics): a platform-side `dataset_schema` rejection on one record is caught and retried with a minimal no-null compliant record, so one bad record can never fail the batch. (3) The Akamai/PerimeterX block-exhaustion path now sets `{ isStatusMessageTerminal: true, level: 'WARNING' }` on its status message and pushes its diagnostic via `safePushData`. Scraping/extraction and the session-rotation/backoff escalation are unchanged.

### 0.7 — 2026-06-11

- Success-rate fix: empty/mismatched user input (no `searchQuery`, `productUrls`, or `itemIds` for the selected mode) now exits SUCCEEDED with a terminal WARNING status telling the user exactly what to supply, instead of calling `Actor.fail`. A user-configuration mistake no longer counts as an actor failure on the 30-day success-rate counter. Scraping, extraction, and the Akamai/PerimeterX session-rotation/backoff escalation are unchanged.

### 0.6 — 2026-06-10

- Success-rate hardening: final Akamai/PerimeterX exhaustion now writes a non-billable diagnostic dataset row, stores RUN\_SUMMARY.stopReason, and exits SUCCEEDED instead of Actor.fail. The schema prefill now requests 5 products instead of 50 so automated health checks stay inside 5 minutes.

### 0.5 — 2026-06-06

- Anti-bot retry escalation to lift the 30-day success rate above 81.8%. A single transient Akamai/PerimeterX block on the first request no longer fails the whole run. Block pages and empty `__NEXT_DATA__` responses now throw Crawlee `SessionError`, which rotates to a FRESH US residential session (different egress IP) on a dedicated `maxSessionRotations` budget (8) — separate from `maxRequestRetries` (raised 4→6). Exponential backoff (~1.5s→30s) between rotated attempts plus `sameDomainDelaySecs: 3` give PerimeterX time to cool off. The honest-fail (`Actor.fail`) is retained as the FINAL fallback only — it now fires after the escalation is exhausted, never as a silent empty-dataset success.

### 0.4 — 2026-06-04

- Category mode removed from input schema. Walmart's `/shop/`, `/browse/`, `/cp/`, and `/search?cat_id=` endpoints either return empty searchResult payloads or get aggressively blocked by Akamai/PerimeterX. Brutal-tested 4 URL variants in v0.3 — none returned usable data. Workaround: use `mode=search` with a category-name keyword (e.g. 'gaming laptop') and filter with `minPrice`/`maxPrice`/`brands`/`minRating`. Internal `category` handler code retained for future iteration when Walmart's category payload shape stabilises.

### 0.3 — 2026-06-04

- Category mode now works on `/browse/...` and `/cp/...` URLs that end in a cat\_id digit chain (e.g. `/browse/electronics/headphones/3944_1101193_447529_1071966`). Auto-converted to `/search?cat_id=…` which uses Walmart's standard search payload and dodges PerimeterX bias against /browse and /cp.
- `/shop/…` URLs without an extractable cat\_id are still attempted as-is but documented as higher-block-rate.

### 0.2 — 2026-06-04

- Search cards now return prices and brands reliably — added 10+ Walmart Next.js price paths (currentPrice/linePrice/priceString/salePrice variants) + string-price parser handling formats like '$24.99', 'From $X', '$X - $Y'.
- Category mode now extracts products from `/shop`, `/browse`, and `/cp` URLs — added `contentLayout.modules[]` fallback for category-page shape.
- itemIds mode rewritten to resolve via search lookup `?q={id}` — Walmart's PerimeterX no longer blocks slug-less `/ip/{id}` URLs into infinite retry loops.
- Cost safety: `maxRequestRetries` 10 → 4, `navigationTimeoutSecs` 60 → 45, `requestHandlerTimeoutSecs` 120 → 90. Caps runaway runs at ~3-4 min instead of full 1h timeout.
- All `productUrl` / `imageUrl` fields returned as absolute `https://www.walmart.com/...` URLs (no more relative paths).
- Live example IDs in input schema (13544111159 / 741224561 — JLab + ONN headphones, both verified live in v0.2 testing).

### 0.1 — 2026-05-13

- Initial release: all-in-one Walmart.com extractor with 4 input modes (search / category / productUrls / itemIds) and optional per-review extraction.
- PlaywrightCrawler (Chromium) with Apify Residential proxy pinned to US, fingerprint pool, session pool with cookie persistence, resource blocking for image/font/css/media.
- Extracts product data from Next.js **NEXT\_DATA** JSON blob — defensive multi-path picker covers Walmart's variant layouts.
- Pay-per-event: actor-start $0.00005, product-extracted $0.003, review-extracted $0.002.
- Silent-failure guard: runs with 0 products and >0 errors fail with actionable status message rather than charging the start fee on an empty dataset.
- RUN\_SUMMARY written to KV at run end with input snapshot, counts, charges, warning samples.