🛒 Walmart Product Scraper

LICENSE

Extract comprehensive product data from Walmart.com at scale. Built with Crawlee and Playwright for reliable, browser-based scraping that handles JavaScript-rendered content, dynamic pricing, and anti-bot protections.

---

✨ Features

• Product Details — Name, SKU/ID, full description, high-res image URLs, seller information
• Price Comparison — Current prices, currency tracking, historical price snapshots via re-runs
• Availability Tracking — Real-time stock status with fulfillment badge capture
• Ratings & Reviews — Star ratings (1–5) and total review count extraction
• Category Navigation — Automatic breadcrumb extraction for taxonomy mapping
• Search & Browse — Start from search queries, category URLs, or a list of product pages
• Price Filtering — Set minPrice and maxPrice to target specific price ranges
• Multilingual Support — Works across Walmart domains (US, Canada, Mexico, and more)
• Review Extraction — Optional review data alongside product info
• Residential Proxy — Built-in Apify proxy with residential group for reliable access
• Anti-blocking — Real browser fingerprints, smart request throttling, and resource blocking for speed

---

📋 Input Parameters

Parameter	Type	Required	Default	Description
startUrls	Array	No	Walmart laptops browse	List of product or category URLs to start scraping
searchQuery	String	No	—	Search term (e.g., "iPhone 15", "Nike shoes")
categoryUrl	String	No	—	Full Walmart category browse URL
maxProducts	Integer	No	100	Maximum products to scrape (0 = unlimited)
includeReviews	Boolean	No	false	Extract review data alongside product info
minPrice	Number	No	—	Minimum price filter
maxPrice	Number	No	—	Maximum price filter
proxyConfiguration	Object	No	Residential proxy	Apify proxy configuration

Note: If no startUrls, searchQuery, or categoryUrl is provided, the scraper defaults to Walmart's laptops category.

---

📤 Output Fields

Each product is saved as a separate dataset item with the following fields:

Field	Type	Description
name	String	Product title
price	String	Current price (formatted)
currency	String	Currency code (e.g., USD, CAD, MXN)
rating	Number	Star rating (0–5)
reviewCount	Number	Number of customer reviews
availability	String	Stock status / fulfillment badge
sku	String	Walmart product ID / SKU
category	String	Full category breadcrumb path
seller	String	Seller name (defaults to "Walmart")
description	String	Product description (truncated to 2000 chars)
imageUrls	Array	Array of high-res product image URLs
url	String	Original product page URL
scrapedAt	String	ISO 8601 timestamp of extraction
error	String	Error message if scraping failed (optional)

---

🚀 Quick Start

Apify Platform

1. Open the Walmart Product Scraper on Apify Store
2. Click Try or Use
3. Configure your input parameters
4. Click Run and wait for results
5. Export data as JSON, CSV, XML, or Excel

Local Development

# Clone and install
git clone https://github.com/your-org/walmart-scraper.git
cd walmart-scraper
npm install

# Run with default input
npx playwright install chromium
node src/main.js

---

🧪 Use Cases

• Price Monitoring — Track price changes over time by running the scraper periodically
• Competitor Analysis — Compare product assortments, pricing, and availability
• Inventory Tracking — Monitor stock levels and out-of-stock patterns
• Market Research — Catalog entire categories with ratings and review counts
• Feed Generation — Build product feeds for comparison shopping engines
• Brand Monitoring — Track authorized and third-party sellers for specific brands

---

🌐 Multilingual Domains

The scraper can target any Walmart regional domain by providing the appropriate URL in startUrls or categoryUrl:

• https://www.walmart.com — United States
• https://www.walmart.ca — Canada
• https://www.walmart.com.mx — Mexico

Use the appropriate proxy group for best results with regional domains.

---

⚙️ Technical Details

• Runtime: Node.js 20 on Apify platform
• Engine: Crawlee PlaywrightCrawler with full browser automation
• Browser: Chromium (headless)
• Proxy: Apify proxy with residential group support
• Concurrency: Auto-scaled based on available resources
• Retry: Automatic retry with exponential backoff for failed requests
• Storage: Apify Dataset with structured output

---

🔒 Error Handling

Failed product pages are still recorded in the dataset with the original URL and an error message, ensuring no data loss. The scraper continues processing remaining items even when individual pages fail.

---

📄 License

This project is licensed under the Apache 2.0 License — see the LICENSE file for details.