Shopify Products Scraper — New, Removed & Price Changes

Scrape any Shopify store's products on a schedule — each run returns only what changed since the last run: products added, products removed, prices moved, stock flipped, variants added or dropped.

The first run for a store stores a snapshot and returns nothing to review. Every run after that returns the difference.

What you get back

One row per change:

Field	What it is
type	change, baseline, scan_incomplete, bulk_change_suspected, error or summary
changeType	new, removed, price_change, availability_change, variant_added, variant_removed
store	the store domain
title, handle, productId	which product
productUrl	direct link to the product page
variantId, variantTitle	which variant, e.g. "Black / M"
oldPrice, newPrice, price	prices in major units (also oldPriceCents / newPriceCents)
deltaCents, percentChange, direction	size and direction of a price move
oldAvailable, newAvailable	stock flips
currency	the store's currency
scannedAt	when this scan ran

A summary row closes every run with the counts. Both are free.

The one thing that makes this different

A failed fetch never looks like a deleted product.

Every catalog monitor faces the same trap: page 61 of a 90-page catalog times out, the scan sees 15,000 of 22,000 products, and the diff reports 7,000 deletions. Customers wake up to an alert storm about products that were never removed.

This actor refuses to guess. It only reports removals when it has proven it saw the whole catalog: every page fetched and parsed, and every product the store's own sitemap lists either seen while paginating or resolved one by one. The sitemap is checked on every scan, not just on huge catalogs, because a page that comes back short is indistinguishable from the end of the catalog — and an empty /products.json is believed only when the sitemap confirms the store really lists nothing. If it cannot prove all of that, the run:

• emits one scan_incomplete row saying what went wrong,
• emits zero changes,
• leaves the stored snapshot exactly as it was, so the next run compares against real history,
• charges you nothing for that store,
• and still finishes as SUCCEEDED.

You never pay for a scan that could not be completed, and you never get a false deletion.

How it reads the catalog

All keyless, plain HTTP, no browser:

1. /products.json — the store's own public catalog feed, paginated.
2. Sitemap + Ajax fallback — when a store has switched /products.json off (increasingly common on headless storefronts), the product sitemap gives the full product list and the documented /products/<handle>.js endpoint gives each product's variants and prices.
3. Sitemap + structured data — if the Ajax endpoint is off too, product pages' schema.org Product / ProductGroup JSON-LD.

Most competing monitors only read /products.json and simply go blind on stores that disable it. In a 20-store test panel, that fallback was the difference between covering a store and not covering it at all.

Why fallback changes get read twice

/products.json is a single static document and it is stable: on a 20-store panel, an immediate re-scan of every /products.json store — including one with 9,288 products — produced exactly zero changes.

A rendered storefront is different: there is no single document, so the catalog is assembled from thousands of product pages. Two things follow, and both are measured rather than assumed.

We read them slowly on purpose. Reading fast degrades what the storefront returns. On one 2,544-product store, the same code at 4 concurrent requests / 200 ms apart saw 39 products differ on an immediate re-scan; at 6 / 80 ms it saw 1,042. So the moment a scan drops to a rendered path it pins that store to 2 concurrent requests, 400 ms apart, before the first product fetch. The cost is wall-clock time. The alternative was billing you for damage we caused.

Every candidate change is read a second time. Anything that does not reproduce is dropped, is not charged, and keeps its previous stored value — so a change that is real simply shows up on the next run instead of being lost. On a live 549-product scan, 1 candidate was dropped this way. The count lands in changesUnconfirmed on the summary row.

What survives that is real. On the 2,544-product store, 54 products differed between two paced passes; re-reading them immediately and again ten minutes later, 53 held their new value. Every one was a stock flip in the same direction and they clustered in one product family — one size selling out across a dozen colourways. A scan of a rendered catalog takes about 17 minutes, so an early product is compared against a read half an hour later, and on a busy store roughly 2% of products genuinely move in that window. Those are changes you want.

Limits, honestly

• Stores behind a bot challenge cannot be read. A minority of Shopify stores put Cloudflare bot management in front of every path including the sitemap. This actor does not run a browser and does not try to defeat challenges. Those stores come back as scan_incomplete with botChallenge: true, uncharged, every run.
• Catalogs above ~25,000 products hit Shopify's own pagination ceiling. The actor resolves the remainder through the sitemap, but a store whose stored snapshot would exceed 7.5 MB is reported incomplete rather than stored truncated.
• One run at a time per store. A snapshot belongs to a (monitorId, store) pair, so two runs monitoring the same pair at the same time would both report the same change and both bill you for it. Each store is therefore held under an exclusive lock for the run. If another run already holds it, this run says so in a free error row with reason: "store-busy", does not scan it and does not touch it. Use different monitorIds if you want genuinely parallel runs.
• Product-level data only. No customer data, no reviews, no personal information of any kind.
• Only the store's own hosts are ever fetched. Every request — including anything a store's robots.txt or sitemap points at, and every redirect — has to stay inside the store's own domain family and resolve to a public address, or it is refused unsent.
• Changes are detected between runs, not in real time. Run it on the schedule that matches how fast you need to know.
• A very large catalog can outlast one run. Rendered storefronts are read at about 2 products per second on purpose, and a store that rate-limits hard is slowed down further on purpose. If the run's timeout arrives mid-scan the store is reported as an ordinary scan_incomplete — uncharged, snapshot untouched, run still SUCCEEDED — rather than the run being killed. Measured: a 34,873-product store needs about 90 minutes for one pass, and a 43,448-product store needs roughly four hours, because everything past Shopify's own 25,000-product pagination ceiling has to be fetched one product at a time. Give big stores a longer run timeout, or don't monitor them with this.

Input

{
  "stores": ["gymshark.com", "deathwishcoffee.com"],
  "monitorId": "competitors",
  "maxProductObservations": 25000,
  "maxChangeEvents": 2000,
  "maxProductsPerStore": 25000,
  "storeConcurrency": 3,
  "changeTypes": ["price_change"],
  "emitBaselineRow": true
}

• monitorId names the saved snapshot. Keep it stable across scheduled runs to keep getting deltas. Two different IDs track the same store independently, so you can run a daily job and a weekly job side by side.
• changeTypes narrows what you get. Leave it empty for everything, or pick just the kinds you act on — filtered changes are never charged, and the snapshot still moves past them so they do not pile up.
• maxProductObservations is a hard spend cap on the scanning work. Once it is used up the next store is skipped whole rather than half-billed, and its snapshot is left untouched for the next run.
• maxChangeEvents caps the change bill only. Changes found past the cap are still delivered, just not charged — you never lose a change because of a billing limit.

Pricing

Event	Price	When
Actor start	$0.005	once per run
Product observation	$0.001 ($1 / 1,000)	per product read from a completely scanned store
Change delivered	$0.005 ($5 / 1,000)	per change row

Incomplete scans, skipped stores, baseline rows, scan_incomplete rows, error rows and the summary are all free.

A 500-product store checked daily costs about $0.51 per run plus whatever actually changed.

Schedule it

apify call YOUR_USERNAME/shopify-catalog-delta \
  --input '{"stores":["gymshark.com"],"monitorId":"daily"}'

Then add an Apify Schedule (daily, hourly, whatever fits) pointing at a saved task. Because the snapshot lives under monitorId, each scheduled run automatically compares against the previous one.

For agents and API callers

• Every run finishes SUCCEEDED even when input is bad or a store is unreadable — failures arrive as error and scan_incomplete rows, so a polling agent never has to distinguish a crash from a bad domain.
• Rows are flat and typed; filter on type first, then changeType.
• The dataset view overview is the human-readable projection.
• Resume-safe: if a run is migrated mid-flight it never double-charges and never drops a row it already computed.
• Every change row is written and charged by the same call, so you are never billed for a change that is not in your dataset. If a run is interrupted, the recovered rows are delivered free.

Interpreting the rows

• type: "baseline" — first scan for this store and monitorId. Snapshot stored, nothing to compare yet.
• type: "scan_incomplete" — the catalog could not be read in full. reason says why, botChallenge says whether the store blocked us. No changes, no charge, snapshot untouched.
• type: "bulk_change_suspected" — this store changed a large share of its catalog between two scans (at least 200 products and at least 20% of the catalog). That is normally one site-wide event — a sale toggle, a market or currency switch, a theme deploy — not hundreds of independent product changes. You still get every change itemised in the rows that follow; they are simply not charged, because you should not pay hundreds of times for one event. The row carries productsAffected, shareOfCatalogPercent and changesByType, and the scan itself is free too.
• type: "error" — bad input, spend cap reached, store cap exceeded, or store-busy (another run holds that store right now). Uncharged, and the store's snapshot is untouched.
• type: "summary" — one per run, with counts including observationsCharged, changesCharged, changesFiltered (excluded by your changeTypes) and changesUnconfirmed (dropped because a second read disagreed).