# Walmart Products Scraper (`scrapyx/walmart-products-scraper`) Actor

Products from Walmart search, category shelves and item pages. Treats PerimeterX's HTTP 200 block page as the block it is, keeps recommendation carousels out of your search results, and never reports Walmart's unstable match count as a total.

- **URL**: https://apify.com/scrapyx/walmart-products-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** E-commerce, Business, Marketing
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.75 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Walmart Products Scraper

Products from **Walmart.com** — keyword search, category shelves and
individual item pages. HTTP-only, no API key, no login, no browser.

### Modes

| Mode | What you get |
| --- | --- |
| `search` | Walmart's own keyword search, walked page by page. |
| `browse` | A category shelf, e.g. `electronics/3944`. |
| `products` | The **full** item document for products you name — long description, full image set, UPC, model number, seller and review histogram. |

### Six upstream quirks it corrects

#### 1. Every block is an **HTTP 200**

There is no 403 anywhere on this host. A blocked request returns:

```
HTTP 200, ~15,561 bytes, `px-captcha` in the body, no __NEXT_DATA__
```

PerimeterX under a 200 status. A scraper that classifies on the status code
sees a clean success for every single block and concludes the shelf is simply
empty — a run that "succeeds" with nothing in it, and a log that says nothing
went wrong. This actor checks for the data layer and the block marker **first**,
and only then looks at the status. `perimeterxBlocksSeen` is published on every
run.

#### 2. The block is per **(TLS profile × surface)**, and one clean response is not a verdict

Full ladder, 14 profiles, same IP, minutes apart:

| Surface | Profiles clean |
| --- | ---: |
| browse page 1 | **all 14** |
| browse `?page=2` | 9 of 14 |
| `/ip/{id}` detail | 3 of 14 |
| `/search?q=` | **1 of 14** |

Re-running the three detail survivors three times each separated the stable
winners from the fluke: `safari18_0` passed once and then failed 3/3, while
`chrome99_android` and `safari17_2_ios` passed 3/3. Then the final matrix —
2 mobile profiles × 4 surfaces × 3 repetitions:

| Profile | browse p1 | browse pN | detail | search |
| --- | --- | --- | --- | --- |
| `chrome99_android` | OK | OK | OK | **blocked 3/3** |
| `safari17_2_ios` | OK | OK | OK | OK |

`safari17_2_ios` is the only profile that walks every surface, so it is not one
option in a rotation — it is the actor. There is deliberately **no rotation
pool**: every alternative is permanently blocked somewhere, so rotating after a
failure would move the run onto a profile that can never succeed there. Retries
re-try the same profile behind a fresh session and a longer backoff.

#### 3. Search results are split across a dozen stacks, and most are **not results**

`?q=laptop` came back as **12 `itemStacks`**:

```
stack0   LIST      'Results for "laptop"'        6 items
stack1   CAROUSEL  '4 stars and above'           6 items
stack3   CAROUSEL  'Shop trending items'        10 items
stack8   CAROUSEL  "Don't miss these savings"   10 items
stack11  LIST      'Results for "laptop"'       25 items
...
```

The CAROUSEL stacks are recommendation widgets. They hold real products, they
are not results for your query, and concatenating everything inflates one page
from 54 rows to 82 while a third of them answer a different question. Only
`layoutEnum == "LIST"` is treated as a result; the rest are counted in
`carouselRowsExcluded` so the discarding is auditable rather than silent.

#### 4. Not every row in a results stack is a product

LIST stacks also carry `AdPlaceholder` and `TileTakeOverProductPlaceholder`
entries — ad slots with no id, no name and no price, filled in client-side.
Skipped and counted as `placeholderRowsSkipped`.

**Sponsored products, by contrast, are kept and flagged.** They sit inline with
organic results and are indistinguishable by position; which of the two you want
is your decision, so every row carries `isSponsored`.

#### 5. Three disagreeing counts, and the obvious one is the worst

One `?q=laptop` response carried, simultaneously:

| Field | Value | What it actually is |
| --- | ---: | --- |
| `aggregatedCount` | 13,208 | matches for the query |
| `count` | 82 | rows across **all** stacks, carousels and ad slots included |
| `gridItemsCount` | 56 | grid rows only |

`count` is the field that reads like a result count and is the least related to
one. All three are published under their own names.

**And `aggregatedCount` is not even stable across pages of the same query.**
Walking one browse shelf and reading it off each page:

```
p1 486,991 | p2 491,788 | p3 900,010 | p5 900,010 | p10 487,608
p15 900,010 | p20 900,008 | p21 900,009 | p22 0
```

It flips between roughly 487k and 900k — an 85% swing on an unchanged query.
It is read **once**, from page 1, published as `aggregatedCountFirstPage` with
`aggregatedCountIsUnstable: true`, and every per-page value ships alongside it
in `aggregatedCountPerPage` so the instability is visible in your data instead
of hidden behind one confident number.

#### 6. `maxPage` over-claims, and disagrees between TLS profiles

The same browse URL told `chrome124` its `maxPage` was **25** and
`safari17_2_ios` that it was **50**. Bisected with the latter, the walk really
ends at page **21**:

```
p20 -> 27 products | p21 -> 17 | p22 -> 0 | p23 -> 0 | p100 -> 0
```

Pages past the end return HTTP 200 with zero products — an honest empty page,
with no wrap back to page 1 and no clamp to the last page. That empty page is
the stop condition; `maxPage` is published for you as
`maxPageClaimedByUpstream` and **never** used to decide when to stop.

#### Bonus: the same item has two incompatible **price** schemas

On a search or browse row:

```
itemPrice   "$278.00"   <- NOT what you pay, despite the name
linePrice   "$258.00"   <- what you pay
wasPrice    "$278.00"   <- a plain string
```

On that same item's own page, the same minute:

```
currentPrice  {"price": 258, "priceString": "$258.00", "currencyUnit": "USD"}
wasPrice      {"price": 278, ...}          <- an OBJECT this time
```

`wasPrice` is a string on one surface and an object on the other. A reader who
takes `itemPrice` for the price overstates this item by $20; one who types
`wasPrice` as a string breaks the moment a detail row arrives. Both shapes
reduce to `currentPrice` / `wasPrice` / `savingsAmount` as plain floats, the raw
object survives as `priceInfo`, and `priceShape` records which schema each row
came from.

`currency` is reported **only** where upstream states it — detail rows carry
`USD`, listing rows carry no currency code anywhere in the payload, and
inventing one is a guess this actor does not make.

### Output

One `SEARCH_SUMMARY` row per run, one `PRODUCT` row per product, one `ERROR`
row per item that could not be fetched.

`PRODUCT` rows carry the upstream object **verbatim**, plus normalised twins:
`itemId`, `productUrl`, `productName`, `brandName`, `productType`,
`modelNumber`, `upc`, `sellerName`, `availability`, `averageRating`,
`reviewCount`, `currentPrice`, `wasPrice`, `savingsAmount`, `currency`,
`isPriceReduced`, `isSponsored`, `resultRank`, `priceShape` and
`metadataSource`.

### Limits

- **There is no reachable total.** Walmart's headline match count is unstable
  and its `maxPage` over-claims; a shelf claiming 487,000 products really ran
  out at page 21 (~600 rows). What you can reach is what the walk returns.
- Listing pages are sequential; `maxConcurrency` applies to detail fetches only.
- A detail miss degrades that one row back to its listing shape — it never
  drops the row and never fails the walk.
- Prices and availability are national defaults. This actor does not set a
  store or a ZIP, so nothing here is store-level inventory.

# Actor input Schema

## `mode` (type: `string`):

search = Walmart's own keyword search. browse = a category shelf. products = full detail for specific items you name.

## `searchTerm` (type: `string`):

For mode='search'. A term that matches nothing returns an honest zero — Walmart does not silently widen the query.

## `browsePath` (type: `string`):

For mode='browse'. Either 'electronics/3944' or the full https://www.walmart.com/browse/... URL. A category id that does not exist honestly returns zero products with an empty title, never the unfiltered catalogue.

## `itemIds` (type: `array`):

For mode='products'. A numeric usItemId (e.g. 16109363845) or any walmart.com/ip/ URL. An item that no longer exists answers HTTP 404 with a null product — read from the payload, not the status.

## `maxResults` (type: `integer`):

Set 0 for unlimited. Note Walmart's own headline match count is NOT a reachable total — the same browse walk reported figures between 487,000 and 900,010 across its own pages and really ran out of products at page 21.

## `fetchFullDetail` (type: `boolean`):

search/browse only — products mode always fetches it. Adds the long description, the full image set, UPC, model number and the review histogram, at one extra request per product. A detail miss degrades that one row back to its listing shape; it never drops the row.

## `maxConcurrency` (type: `integer`):

Detail fetches in flight at once. Listing pages are sequential. Keep this low — this host scores request bursts.

## `minRequestInterval` (type: `integer`):

Politeness pacing shared across all workers. 0 uses the built-in default of 1.2s. Raising this is the correct response to PerimeterX blocks — changing the TLS profile is not, since only one profile is clean on every surface.

## `proxyConfiguration` (type: `object`):

Apify Proxy on the shared datacenter pool. This is the default because it is included in your plan at no extra cost and this target works through it. If you start seeing blocks, challenges or empty results, switch the group here to Residential -- it uses real consumer IPs and gets through more, but Apify bills residential traffic per gigabyte, so leave it off unless you need it.

## Actor input object example

```json
{
  "mode": "search",
  "searchTerm": "laptop",
  "browsePath": "electronics/3944",
  "itemIds": [
    "16109363845",
    "https://www.walmart.com/ip/5689919121"
  ],
  "maxResults": 100,
  "fetchFullDetail": false,
  "maxConcurrency": 3,
  "minRequestInterval": 0,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/walmart-products-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/walmart-products-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call scrapyx/walmart-products-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/walmart-products-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/SbecdKKy4PLxHx1Cw/builds/l9oqPXYU2xbTb9Z5N/openapi.json