Shopify Products Scraper & Price Monitor

Scrape the public /products.json catalog of any number of Shopify stores and turn it into clean, structured data — then keep watching it. This Shopify scraper is built for price monitoring, catalog building and competitor research: run it once for a snapshot, or run it on a schedule to get price drop alerts, new‑product alerts and inventory tracker signals across a whole group of stores.

Unlike single‑store, one‑shot scrapers, this actor is a multi‑store price monitor: point it at 1 or 100 Shopify storefronts, and in monitor mode it diffs every run against the last one and emits change events (price changed, new product, product removed, availability changed).

What it does

• Reads only the public /products.json endpoint that Shopify serves by design — no login, no headless browser, no private data.
• Handles pagination automatically (?limit=250&page=n) up to an optional per‑store cap.
• Checks robots.txt for every store before scraping and skips any store that disallows the endpoint.
• Normalizes each product into a flat item with computed min/max price, variant detail, and availability counts.
• In monitor mode, compares against the previous run and outputs price‑change / new / removed / availability events.

Use cases

• Competitor price monitoring — track a group of rival Shopify stores and get alerted the moment a price drops.
• Product research / sourcing — pull entire catalogs (title, vendor, type, tags, variants, price range) for analysis.
• Catalog / feed building — keep an up‑to‑date structured mirror of a store's public catalog.
• Inventory tracker — watch which variants flip in and out of stock over time.

Input

Field	Type	Default	Description
stores	string[]	— (required)	Domains or URLs, e.g. allbirds.com or https://www.gymshark.com.
mode	snapshot | monitor	snapshot	snapshot = one‑off dump. monitor = also diff vs. the last run.
maxProductsPerStore	integer	0	Cap per store. 0 = all.
includeVariantDetail	boolean	true	Include the per‑variant array on each product.
throttleMs	integer	1500	Minimum delay between requests to the same store (stores run in parallel).
proxyConfiguration	object	{}	Optional Apify proxy. Empty = direct connection.

{
  "stores": ["allbirds.com", "https://www.gymshark.com"],
  "mode": "monitor",
  "maxProductsPerStore": 0,
  "includeVariantDetail": true,
  "throttleMs": 1500
}

Output

Product item (one per product; real sample, trimmed):

{
  "type": "product",
  "store": "www.deathwishcoffee.com",
  "product_id": 7469535264823,
  "handle": "all-american-tank",
  "title": "All-American Tank",
  "vendor": "Planet Apparel",
  "product_type": "Apparel",
  "tags": ["Americana", "Apparel", "Men", "Unisex"],
  "created_at": "2026-06-15T13:35:58-04:00",
  "updated_at": "2026-07-05T04:17:35-04:00",
  "published_at": "2026-07-04T07:17:47-04:00",
  "price_min": 30,
  "price_max": 30,
  "images_count": 6,
  "available_count": 5,
  "url": "https://www.deathwishcoffee.com/products/all-american-tank",
  "variants": [
    { "id": 42272775372855, "sku": "UTRWBT03", "title": "M", "price": 30, "compare_at_price": null, "available": true, "options": ["M"] }
  ]
}

A small store_status item is also emitted per store (ok / skipped_robots / endpoint_unavailable / rate_limited / error) so failures are visible in the dataset without breaking the run.

Monitor mode (price & stock alerts)

In monitor mode the actor stores a compact snapshot of each store in a named key‑value store and, on the next run, compares the fresh catalog against it. Every difference becomes a change event item ("type": "change_event"):

{ "type": "change_event", "event": "price_changed", "store": "allbirds.com",
  "product_id": 7205181653072, "title": "...",
  "before": { "price_min": 185, "price_max": 185 },
  "after":  { "price_min": 135, "price_max": 135 }, "ts": "2026-07-05T..." }

Event types: price_changed, new_product, product_removed, availability_changed. Schedule the actor (e.g. hourly/daily) and wire the change‑event stream into your alerting to get price‑drop alerts and stock notifications. The first monitor run establishes a baseline and emits no events.

Compliance

• Public data only. The actor requests exclusively /robots.txt and the public /products.json endpoint that Shopify serves by design. It never touches /admin, carts, checkout, orders, reviews or any customer data.
• No personal data is collected — only product/catalog metadata.
• robots.txt is respected: each store is checked first and skipped if /products.json is disallowed for our user‑agent.
• No product descriptions (body_html) are stored — only structured metadata (title, vendor, tags, prices, variants), to avoid copying store copy.
• Descriptive user‑agent identifies the crawler and its narrow scope.
• You are responsible for your use. Review the target store's Terms of Service and applicable laws for your jurisdiction and use case before running at scale.

FAQ

Does this use the Shopify Admin API or need an API key? No. It reads the public storefront /products.json, so no keys, tokens or login are required.

Will it work on any store? Only stores running Shopify that keep /products.json public. Stores that disable the endpoint (or disallow it in robots.txt) are reported as endpoint_unavailable / skipped_robots and skipped — other stores in the same run are unaffected.

How do I get only price changes? Run in monitor mode on a schedule and filter dataset items to type = change_event, event = price_changed.

Can I limit how much it scrapes? Yes — set maxProductsPerStore. Requests to a single store are throttled by throttleMs; different stores run in parallel.

Does it handle rate limiting? Yes. 429 responses are retried with exponential backoff (2s/4s/8s/16s) before the store is marked rate_limited; transient network errors are retried too.

Local development

npm install && npm run build
apify run -p     # local run; -p purges ./storage first (input: storage/key_value_stores/default/INPUT.json)
npm test         # offline unit tests, no network

Note (Apify SDK v3): the local storage location is controlled by CRAWLEE_STORAGE_DIR — the legacy APIFY_LOCAL_STORAGE_DIR is ignored. Re-running into a non-empty ./storage can collide with the previous run's dataset; either purge (apify run -p) or isolate each run, e.g. CRAWLEE_STORAGE_DIR=$PWD/storage-run2 node dist/main.js.