# Shopify Products Scraper — New, Removed & Price Changes (`kaz_kakyo/shopify-catalog`) Actor

Scrape any Shopify store's full product catalog and get only what changed: new products, removed products, price moves, stock flips, variant adds and drops. Each scheduled run returns the delta against the last. Reads the store's public catalog feeds; sitemap fallback if products.json is off.

- **URL**: https://apify.com/kaz\_kakyo/shopify-catalog.md
- **Developed by:** [Heim AI](https://apify.com/kaz_kakyo) (community)
- **Categories:** E-commerce, Automation, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 change delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Shopify Products Scraper — New, Removed & Price Changes

Scrape any Shopify store's products on a schedule — each run returns **only what changed since the last run**: products added, products removed, prices moved, stock flipped, variants added or dropped.

The first run for a store stores a snapshot and returns nothing to review. Every run after that returns the difference.

### What you get back

One row per change:

| Field | What it is |
|---|---|
| `type` | `change`, `baseline`, `scan_incomplete`, `bulk_change_suspected`, `error` or `summary` |
| `changeType` | `new`, `removed`, `price_change`, `availability_change`, `variant_added`, `variant_removed` |
| `store` | the store domain |
| `title`, `handle`, `productId` | which product |
| `productUrl` | direct link to the product page |
| `variantId`, `variantTitle` | which variant, e.g. "Black / M" |
| `oldPrice`, `newPrice`, `price` | prices in major units (also `oldPriceCents` / `newPriceCents`) |
| `deltaCents`, `percentChange`, `direction` | size and direction of a price move |
| `oldAvailable`, `newAvailable` | stock flips |
| `currency` | the store's currency |
| `scannedAt` | when this scan ran |

A `summary` row closes every run with the counts. Both are free.

### The one thing that makes this different

**A failed fetch never looks like a deleted product.**

Every catalog monitor faces the same trap: page 61 of a 90-page catalog times out, the scan sees 15,000 of 22,000 products, and the diff reports 7,000 deletions. Customers wake up to an alert storm about products that were never removed.

This actor refuses to guess. It only reports removals when it has **proven** it saw the whole catalog: every page fetched and parsed, **and** every product the store's own sitemap lists either seen while paginating or resolved one by one. The sitemap is checked on every scan, not just on huge catalogs, because a page that comes back short is indistinguishable from the end of the catalog — and an empty `/products.json` is believed only when the sitemap confirms the store really lists nothing. If it cannot prove all of that, the run:

- emits one `scan_incomplete` row saying what went wrong,
- emits **zero** changes,
- leaves the stored snapshot **exactly** as it was, so the next run compares against real history,
- charges you **nothing** for that store,
- and still finishes as **SUCCEEDED**.

You never pay for a scan that could not be completed, and you never get a false deletion.

### How it reads the catalog

All keyless, plain HTTP, no browser:

1. **`/products.json`** — the store's own public catalog feed, paginated.
2. **Sitemap + Ajax fallback** — when a store has switched `/products.json` off (increasingly common on headless storefronts), the product sitemap gives the full product list and the documented `/products/<handle>.js` endpoint gives each product's variants and prices.
3. **Sitemap + structured data** — if the Ajax endpoint is off too, product pages' schema.org `Product` / `ProductGroup` JSON-LD.

Most competing monitors only read `/products.json` and simply go blind on stores that disable it. In a 20-store test panel, that fallback was the difference between covering a store and not covering it at all.

#### Why fallback changes get read twice

`/products.json` is a single static document and it is stable: on a 20-store panel, an immediate re-scan of every `/products.json` store — including one with 9,288 products — produced exactly zero changes.

A rendered storefront is different: there is no single document, so the catalog is assembled from thousands of product pages. Two things follow, and both are measured rather than assumed.

**We read them slowly on purpose.** Reading fast degrades what the storefront returns. On one 2,544-product store, the same code at 4 concurrent requests / 200 ms apart saw 39 products differ on an immediate re-scan; at 6 / 80 ms it saw 1,042. So the moment a scan drops to a rendered path it pins that store to 2 concurrent requests, 400 ms apart, before the first product fetch. The cost is wall-clock time. The alternative was billing you for damage we caused.

**Every candidate change is read a second time.** Anything that does not reproduce is dropped, is **not charged**, and keeps its previous stored value — so a change that is real simply shows up on the next run instead of being lost. On a live 549-product scan, 1 candidate was dropped this way. The count lands in `changesUnconfirmed` on the summary row.

What survives that is real. On the 2,544-product store, 54 products differed between two paced passes; re-reading them immediately and again ten minutes later, 53 held their new value. Every one was a stock flip in the same direction and they clustered in one product family — one size selling out across a dozen colourways. A scan of a rendered catalog takes about 17 minutes, so an early product is compared against a read half an hour later, and on a busy store roughly 2% of products genuinely move in that window. Those are changes you want.

### Limits, honestly

- **Stores behind a bot challenge cannot be read.** A minority of Shopify stores put Cloudflare bot management in front of every path including the sitemap. This actor does not run a browser and does not try to defeat challenges. Those stores come back as `scan_incomplete` with `botChallenge: true`, uncharged, every run.
- **Catalogs above ~25,000 products** hit Shopify's own pagination ceiling. The actor resolves the remainder through the sitemap, but a store whose stored snapshot would exceed 7.5 MB is reported incomplete rather than stored truncated.
- **One run at a time per store.** A snapshot belongs to a `(monitorId, store)` pair, so two runs monitoring the same pair at the same time would both report the same change and both bill you for it. Each store is therefore held under an exclusive lock for the run. If another run already holds it, this run says so in a free `error` row with `reason: "store-busy"`, does not scan it and does not touch it. Use different `monitorId`s if you want genuinely parallel runs.
- **Product-level data only.** No customer data, no reviews, no personal information of any kind.
- **Only the store's own hosts are ever fetched.** Every request — including anything a store's `robots.txt` or sitemap points at, and every redirect — has to stay inside the store's own domain family and resolve to a public address, or it is refused unsent.
- **Changes are detected between runs**, not in real time. Run it on the schedule that matches how fast you need to know.
- **A very large catalog can outlast one run.** Rendered storefronts are read at about 2 products per second on purpose, and a store that rate-limits hard is slowed down further on purpose. If the run's timeout arrives mid-scan the store is reported as an ordinary `scan_incomplete` — uncharged, snapshot untouched, run still SUCCEEDED — rather than the run being killed. Measured: a 34,873-product store needs about 90 minutes for one pass, and a 43,448-product store needs roughly four hours, because everything past Shopify's own 25,000-product pagination ceiling has to be fetched one product at a time. Give big stores a longer run timeout, or don't monitor them with this.

### Input

```json
{
  "stores": ["gymshark.com", "deathwishcoffee.com"],
  "monitorId": "competitors",
  "maxProductObservations": 25000,
  "maxChangeEvents": 2000,
  "maxProductsPerStore": 25000,
  "storeConcurrency": 3,
  "changeTypes": ["price_change"],
  "emitBaselineRow": true
}
```

- **`monitorId`** names the saved snapshot. Keep it stable across scheduled runs to keep getting deltas. Two different IDs track the same store independently, so you can run a daily job and a weekly job side by side.
- **`changeTypes`** narrows what you get. Leave it empty for everything, or pick just the kinds you act on — filtered changes are never charged, and the snapshot still moves past them so they do not pile up.
- **`maxProductObservations`** is a hard spend cap on the scanning work. Once it is used up the next store is skipped whole rather than half-billed, and its snapshot is left untouched for the next run.
- **`maxChangeEvents`** caps the change bill only. Changes found past the cap are still delivered, just not charged — you never lose a change because of a billing limit.

### Pricing

| Event | Price | When |
|---|---|---|
| Actor start | $0.005 | once per run |
| Product observation | $0.001 ($1 / 1,000) | per product read from a **completely scanned** store |
| Change delivered | $0.005 ($5 / 1,000) | per change row |

Incomplete scans, skipped stores, baseline rows, `scan_incomplete` rows, error rows and the summary are all free.

A 500-product store checked daily costs about $0.51 per run plus whatever actually changed.

### Schedule it

```bash
apify call YOUR_USERNAME/shopify-catalog-delta \
  --input '{"stores":["gymshark.com"],"monitorId":"daily"}'
```

Then add an Apify Schedule (daily, hourly, whatever fits) pointing at a saved task. Because the snapshot lives under `monitorId`, each scheduled run automatically compares against the previous one.

### For agents and API callers

- Every run finishes **SUCCEEDED** even when input is bad or a store is unreadable — failures arrive as `error` and `scan_incomplete` rows, so a polling agent never has to distinguish a crash from a bad domain.
- Rows are flat and typed; filter on `type` first, then `changeType`.
- The dataset view `overview` is the human-readable projection.
- Resume-safe: if a run is migrated mid-flight it never double-charges and never drops a row it already computed.
- Every change row is written and charged by the same call, so you are never billed for a change that is not in your dataset. If a run is interrupted, the recovered rows are delivered **free**.

### Interpreting the rows

- `type: "baseline"` — first scan for this store and `monitorId`. Snapshot stored, nothing to compare yet.
- `type: "scan_incomplete"` — the catalog could not be read in full. `reason` says why, `botChallenge` says whether the store blocked us. No changes, no charge, snapshot untouched.
- `type: "bulk_change_suspected"` — this store changed a large share of its catalog between two scans (at least 200 products and at least 20% of the catalog). That is normally one site-wide event — a sale toggle, a market or currency switch, a theme deploy — not hundreds of independent product changes. You still get every change itemised in the rows that follow; they are simply **not charged**, because you should not pay hundreds of times for one event. The row carries `productsAffected`, `shareOfCatalogPercent` and `changesByType`, and the scan itself is free too.
- `type: "error"` — bad input, spend cap reached, store cap exceeded, or `store-busy` (another run holds that store right now). Uncharged, and the store's snapshot is untouched.
- `type: "summary"` — one per run, with counts including `observationsCharged`, `changesCharged`, `changesFiltered` (excluded by your `changeTypes`) and `changesUnconfirmed` (dropped because a second read disagreed).

# Actor input Schema

## `stores` (type: `array`):

Store domains to monitor, e.g. gymshark.com. One row per store; up to 50 per run. No https://, no path.

## `monitorId` (type: `string`):

Names the saved snapshot this run compares against. Keep it the same across scheduled runs to keep getting deltas; change it to start a fresh baseline. Separate IDs never see each other's snapshots.

## `maxProductObservations` (type: `integer`):

Hard ceiling on billable product observations for this run, at $1 per 1,000. A store is skipped rather than started once the ceiling is used up, so a big catalog can never surprise you.

## `maxChangeEvents` (type: `integer`):

Hard ceiling on billable change rows for this run, at $5 per 1,000. Changes found beyond the ceiling are still delivered, just not charged.

## `maxProductsPerStore` (type: `integer`):

Stop scanning a store past this many products. A store that overruns it is reported as an incomplete scan — no changes, no charge, snapshot untouched.

## `storeConcurrency` (type: `integer`):

How many stores to scan in parallel. Lower it if a store starts rate-limiting you.

## `changeTypes` (type: `array`):

Only report these kinds of change. Leave empty for all of them. Anything filtered out is never charged, and the snapshot still moves forward so you do not get it again later.

## `emitBaselineRow` (type: `boolean`):

On the first run for a store there is nothing to compare against. Leave this on to get one free row confirming the snapshot was stored.

## Actor input object example

```json
{
  "stores": [
    "deathwishcoffee.com",
    "glossier.com"
  ],
  "monitorId": "default",
  "maxProductObservations": 25000,
  "maxChangeEvents": 2000,
  "maxProductsPerStore": 25000,
  "storeConcurrency": 3,
  "changeTypes": [],
  "emitBaselineRow": true
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "stores": [
        "deathwishcoffee.com",
        "glossier.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("kaz_kakyo/shopify-catalog").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "stores": [
        "deathwishcoffee.com",
        "glossier.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("kaz_kakyo/shopify-catalog").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "stores": [
    "deathwishcoffee.com",
    "glossier.com"
  ]
}' |
apify call kaz_kakyo/shopify-catalog --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kaz_kakyo/shopify-catalog"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Sj1ay2b3ciEhs1h0W/builds/EwXskbUbPC8SuQbMF/openapi.json