# Changelog of Walmart Products Scraper & Price Tracker (`myth84/walmart-search-scraper`) Actor

- **URL**: https://apify.com/myth84/walmart-search-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/myth84/walmart-search-scraper.md

### Changelog

History of `walmart-search-scraper` (folder `actors/walmart-search`). Versions are the Apify build numbers.

#### Unreleased (agent-facing polish: field order, first screen)

- Brief `docs/cloud-tasks/2026-10-24_studio-agent-facing-polish.md` (2026-10-24): what an AI agent reads first. No change to the code, the defaults, the prices or the output.
- **Input field order** (the Apify MCP search card lists the first 20 fields): `maxPages` moves to *Advanced*. Nothing renamed or removed.
- **README first screen**: under the title, "Verified daily by an automated run on Apify with the default input", the Free-plan and Business-plan prices per 1,000 results of the pricing file in force and the $0.00005 start fee per run (Apify's minimum); a one-line review request before the legal note. A test binds these figures to the pricing file and forbids "to confirm" and "no start fee" in the input schema and the README.

#### Unreleased (price monitoring and price-tracker Store texts)

- **Price monitoring** (brief `docs/cloud-tasks/2026-10-21_walmart-search-extend-monitor.md`, 2026-10-21, modelled on the Amazon and eBay Actors, same option and column names): `monitorPrices` (off by default), `onlyChanges` (off by default) and `monitorStoreName` (default `walmart-price-monitor`, letters, digits and dashes, lower-cased); previous prices kept in the customer's named key-value store, one record (`walmart-monitor/1`) per search identity: keywords, or the customer URL without its tracking parameters (`utm_*`, `ath*`, `povid`, `typeahead`… ; page 1, our sort and price range on top, parameters sorted, no fragment), site, language, sort and price range; a product is identified by its Walmart product id (`usItemId`) inside the record, so the identity of a result is product id + site (hence currency). `changeType` / `previousPrice` / `priceDelta` / `priceDeltaPercent` filled (price change only in the same currency); with `onlyChanges` only the new products and the price changes are pushed, hence charged; a monitoring run that compared products and found no change succeeds with an empty dataset (`isEmptyRunFailure`). `RUN_STATS.monitor` (searches, baseline, compared, new, price changes, unchanged, written, records written and skipped), `RUN_STATS.monitorBaseline`, counters `monitorCompared`, `monitorNew`, `monitorPriceChanged`, `monitorUnchanged`, `monitorDuplicatesNotWritten` in `RUN_STATS.extra`.
- **Same rules as Amazon**: the monitoring pass finds the search of a row through the row itself; whether a search ended cleanly is decided from its own pages (last page requested = last page parsed, no row refused by the charger, stopped by its last page, its cap, `maxPages` or two pages without anything new rather than by the run's budget); in monitoring mode deduplication is per search (`<record key>|<usItemId>`) and the dataset holds a product once per site (`monitorDuplicatesNotWritten`); a product shown without a price keeps its last known price; an unreadable store fails the run before any request, an unwritable one after the crawl, with an explicit `WalmartMonitorStoreError` message; the same search given twice writes one record.
- **Deviation from the Amazon and eBay crawlers**: a search that fills `maxResults` on the page where the runner's charger reaches its run cap (`maxResults` × searches) ended by its own cap, so its record is written. The charger then reports `limitReached` and the generic crawler stops before `isExhausted` is asked again; counting that stop as "the run budget ran out" would never record the reference of a first run with more products than `maxResults` (checked on the Amazon crawler with the same fake charger: `recordsWritten: 0`).
- **Monitoring off: output unchanged.** The charger receives the rows directly; checked once against the 0.1 crawler on every fixture and eight scenarios (three pages, mixed searches on walmart.com, walmart.ca in English and French and a category URL, "Robot or human?" then served, 429 then served, always blocked, sponsored skipped, per-search cap, `maxRequests`): identical rows, pushes, log lines, pauses, requests (URL, proxy, headers, transport) and statistics, and kept as tests (rows equal to the normalizer's on a fixture page, identical output with `monitor: null`). `RUN_STATS` gains the zero counters and `monitor: null`.
- **Store texts on the "walmart price tracker" niche** (`docs/research/2026-10-21_recherches-sans-reponse.md`: 28 results, 24 users at most in the top 10 on 2026-10-21): title and SEO title **Walmart Products Scraper & Price Tracker**, README H2 with the same words; description and SEO description say "Walmart price tracker" and price drops; README opening with "Walmart price tracker", "Walmart price monitor" and "price drop alerts"; `searchSynonyms` gain "Walmart price tracker" and "Walmart price monitor". README: *Price monitoring* section (scheduled input, `price_changed` row computed from the README example row by the code, rules), input table, *Changes* view, cost and limits notes, AI-assistant prompt and MCP block for price tracking; the FAQ keeps 10 entries (freshness merged into the monitoring entry).
- Input form section *Price monitoring*; dataset view **Changes**; monitoring columns described in `dataset-fields.json`; example tasks `walmart-wireless-earbuds-price-drops-us` and `walmart-air-fryer-price-monitor-us` (10 in total, the platform limit); tests on the monitor module, the crawler pass (reference run, changes only, price drop and new product, every product, cap, cap reached with the charger's run cap, missing price, separate searches by site, language, sort and price range, shared products, same search twice, fail-closed cases, per-search decision, unknown record, unreadable and unwritable store), the input and the Store files.

#### Unreleased (quality score: input documented for AI agents)

- Brief `docs/cloud-tasks/2026-10-20_studio-quality-score-insights.md` (2026-10-21): the input schema passes the shared checks of `@scraper-studio/core/testing` (`runInputQualityChecks`, `runSearchTextChecks`), the rules of Apify's quality score and of what an AI agent receives through the Apify MCP server.
- Descriptions rewritten (default stated with its reason, example of the format for free text): `searchQueries` (the default only serves runs started without any keyword or URL), `startUrls` (shortened under the 500 characters the MCP server keeps, example URL, default empty), `site`, `language`, `maxResults` (says the form prefill), `sortBy`, `priceMin`, `priceMax` (examples, `priceMax` at least `priceMin`), `includeSponsored`, `maxPages`, `proxyConfiguration`, `maxRequests`, `delayMs`, `debugStoreHtml`.
- Bounds added to the form: `maximum` 1,000,000,000 on `priceMin` and `priceMax` (the code accepts any non-negative number; no realistic price reaches the bound).
- `maxResults` is prefilled with 20 in the form (the default stays 100): the prefilled input Apify tests daily is exactly `inputs/daily-check.json` (20 products, wireless earbuds, walmart.com), which is unchanged, so the live daily-check task needs no update.
- `store.json`: `searchSynonyms` (Walmart scraper, Walmart prices, Walmart search results, Walmart.ca), all already within the first 3,000 README characters; SEO description now says "Walmart products" ("Scrape Walmart products from Walmart.com & Walmart.ca search: …", 151 characters; "sponsored flag" dropped for length). README input table: prefill of `maxResults`, example values for `priceMin` and `priceMax`.
- **Example task** `walmart-earbuds-category-us`: SEO title rewritten as the long-tail search it targets ("Walmart earbuds category – prices, ratings & stock"); published with `scripts/publish-tasks.mjs` at the publication of the Actor.
- No code change: rows, logs and charges are identical.

#### Unreleased (README visibility)

- **README first screen and links** (brief `docs/cloud-tasks/2026-10-20_studio-readme-visibility-new-actors.md`, 2026-10-21), same layout as the public Actors (PR #50). The Store search indexes only the first ~3,000 README characters, so they now hold: an opening sentence with the phrase agents search ("Walmart products scraper", "Walmart scraper", price tracking with scheduled runs, competitor price monitoring, price comparison), a "No login, no cookies, no personal data" line, the Free-plan price in figures in *How much* ($2.40 per 1,000 products; Apify's $5 monthly credit covers about 2,083), computed from `pricing/2026-10-20_pay-per-event.autopilot.json` (Free tier of the primary event, one `apify-actor-start` event deducted), and a real output snippet (12 key columns of the README example row). Further down: *Use it from your AI assistant* (MCP URL `https://mcp.apify.com?tools=myth84/walmart-search-scraper` and a ready-to-copy prompt with a minimal input), *Alternatives and switching* (official API, home-made scraper, input-name mapping table, no competitor named), *More scrapers by Flatrow Data Studio* (Store links to the e-commerce Actors already public) with an invitation to rate the Actor (no counterpart), and the non-affiliation line at the very bottom. The cost section and the MCP cost bullet quote the same figures; the line 1 heading is unchanged. Checked by `runStoreChecks` (`readmeVisibility`). No code change: rows, logs and charges are identical.

#### 0.1 — 2026-10-20

- First version, written offline on the committed fixtures of the audit (`docs/audits/walmart.md`, brief `docs/cloud-tasks/2026-10-20_walmart-build.md`, 8 pages captured by the probe on walmart.com and walmart.ca plus the two block pages): Next.js payload `<script id="__NEXT_DATA__">` → `props.pageProps.initialData.searchResult`, located with a regular expression and handed to `JSON.parse` as is (pages of up to 1.8 MB); only the "Results for …" / "Résultats pour « … »" stacks are read (stack title or `meta.titleKey === 'resultsFor'`), and in them only the `Product` items, in display order; recommendation stacks ("4 stars and above", "Don't miss these savings", "Shop trending items"…) and ad tiles are skipped (`RUN_STATS.extra.stacksSkipped`); a page without any result stack is read whole (`stackFallbacks`). Pages without the payload (block page, 429 waiting room, department page `/cp/`) raise `WalmartParseError`; a page whose `searchResult` declares zero results is a valid empty page.
- **Pagination**: `page=N` (no parameter on page 1, as validated) until `paginationV2.maxPage` (Walmart's `hasMorePages` is false before the last page and is not read), `maxResults` (default 100, at most 2,000), `maxPages` (default 20, at most 50) or two pages without a new product; global deduplication by `usItemId` (`duplicatesSkipped`); `searchRank` is the position counted across the result stacks of every page, sponsored products included.
- **Transport**: `impit` with the iOS 18 Safari profile on the **Apify datacenter tier by default** (`proxyConfiguration` default `{ "useApifyProxy": true }`, 25/26 pages served in the audit; got-scraping and the Chrome profile are refused); the generic crawler's `Accept` header is removed (`debugSendAccept` sends it), `Accept-Language` follows each URL (`fr-CA` on walmart.ca/fr). The residential tier works too (2/2 in the audit) and stays an option: about 0.04 $ per 1,000 products of proxy traffic instead of about 0.01 $ of compute, both included in the price.
- **Block detection without touching the core**: every served Walmart page references PerimeterX (`*.perimeterx.net` in its CSP, `_pxAppId`) within the 64 KB scanned by `detectBlock`, whose strong markers `perimeterx` / `_pxhc` therefore class it as a challenge. The Actor's `fetch` wrapper calls the client with `maxRetries: 0` and re-reads such a verdict with the two markers blanked out: a 2xx page carrying `id="__NEXT_DATA__"` and `"searchResult"` and none of Walmart's real block signs (`Robot or human?`, `Verify Your Identity`, `Verifica tu identidad`, `/blocked?url=` in the body or the final URL, 429 `/waiting-hallway`) is served (`pxFalsePositives`); otherwise the recomputed verdict is kept (a 429 page is reported as `http-429` with its `Retry-After`).
- **Unusable pages**: on a 429 the wrapper waits `Retry-After` (else 10 s, at most 60 s; `rateLimited`); an unusable page is retried at once on a fresh session (new proxy session, new cookie jar, kept for the next pages of the search; `sessionRetries`), then the generic crawler rotates once more and gives up the search after the third unusable answer in a row. The wrapper's extra attempts are counted in `requests`, the bytes and `blocked`, and respect `maxRequests`. A proxy refusal (407, `ProxyAuthRequired`, `ProxyTunnelError`) is retried once on a fresh session, then fails the run with an explicit message (`WalmartProxyConfigurationError`). `delayMs` defaults to 2,500 ms between two pages of a search (1 × 429 on 28 requests at 1.5 s in the audit, 0 on 22 at 2.5 s).
- Output schema `ecommerce-product/1.1`: `price` from the numeric `price` (every layout), else `priceInfo.linePrice` or the price lines; `originalPrice` from `priceInfo.wasPrice` (walmart.ca, "$89.99", "149,99 $") or, on walmart.com where the `priceInfo` strings are empty, from `priceInfo.priceDetails.priceLines` `COMPARISON` / `WAS_PRICE`; `discountPercent` computed to one decimal; `shippingCost` from `shipPrice`, the `SHIPPING` price line or `shippingFee` (walmart.ca marketplace items); `shippingFree` true only on an unconditional "Free shipping" / "Free delivery" / "Livraison gratuite" badge, false when a shipping price is shown, null otherwise; `deliveryEstimate` from the first fulfillment badge group, else the first delivery date of `fulfillmentSummary` (ISO timestamp); `inStock` from `availabilityStatusV2`, `isOutOfStock`, `availabilityStatus`; `rating` null for products without review; `categoryPath` without "Home Page" (walmart.com) or from `category.path[].name` (walmart.ca); `imageUrl` without `odnHeight` / `odnWidth` / `odnBg`; `url` = site origin + `canonicalUrl` without its query string; `variantId` = `id` for `VARIANT` items; `isBestSeller` from the `BESTSELLER` badge; flat `attributes` (`condition`, `classType`, `catalogProductType`, `fulfillmentType`, `badges`, `variants`, `salesUnit`, `unitPrice`, `additionalOfferCount`, `categoryPathId`, `priceFrom`, `priceMax`, `savings`, `walmartPlusPrice`, `subscriptionPrice`, `socialProof`, `boughtRecently`).
- **Seller and personal data**: `storeType` `platform` (`sellerType` INTERNAL or seller "Walmart" / "Walmart.com", `storeName` "Walmart"), `business` (`sellerType` EXTERNAL: Walmart Marketplace admits registered businesses only, `storeName` = seller name) or `unknown` (store columns cleared by `sanitizeStoreFields`); `storeId` from `catalogSellerId`; `sellerId`, `displayName`, descriptions and reviews are never read; `stripPersonalData` on every row. The Walmart+ price is exported as `walmartPlusPrice`: a `memberPrice…` key would be removed by the privacy filter (person word "member").
- **Decisions**: `sortBy` values `price_high`, `best_seller`, `rating_high`, `new` and the price filter on walmart.ca or on category pages follow Walmart's public URL conventions and are marked "not verified" in the input form, the README and the example tasks until the first runs; a URL's own domain and `/en` / `/fr` prefix decide its site and language; walmart.com.mx URLs are refused with an explicit message; `language: "fr"` requires `site: "CA"`. `store.json › seoDescription` is shortened to 153 characters (the brief's text had 167, above the 155 accepted by `scripts/release-actor.mjs`). Following the Store-text rule of 2026-10-20 (`docs/cloud-tasks/2026-10-20_studio-promote-store-texts.md`), the README names no proxy tier nor anti-bot detail: the proxy guidance lives in the input form (section *Proxy*) and in this changelog.
- Known limits (audit §9–10): 26 requests only behind the 96 % success rate (to confirm on ≥ 100 requests at the control run), national prices without store or postal code, about 500 unique products per search, `storeType` / `storeName` not observable in the fixtures (seller keys removed at capture time), memory of 256 MB to confirm on pages of 1.8 MB.
- Store readiness: README, input schema, dataset schema with Overview / Prices / Availability / Ranking views, output schema, 8 example tasks (default proxy, at most 60 products, unverified sorts said in their description), consistency tests (README ⇔ schemas ⇔ code, Ajv replay of the 363 fixture rows), crawler tests on a fake HTTP client (transport, headers, pagination to `maxPage`, deduplication, rotation on "Robot or human?", wait on 429, PerimeterX false positive, `maxRequests`, proxy error), seller and injection tests.