# Shopify Products Scraper (`scrapyx/shopify-products-scraper`) Actor

Products, variants and collections from any Shopify store. Checks each store's own robots.txt before the first request, and fetches the currency from meta.json - the products endpoint carries prices with no currency anywhere in the payload.

- **URL**: https://apify.com/scrapyx/shopify-products-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** E-commerce, Business, Marketing
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.75 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Shopify Products Scraper

Products, variants and collections from **any** Shopify store, via the public
storefront JSON every store exposes on its own domain. No login, no API key,
no app install, no browser.

### It checks each store's robots.txt before it fetches a single product

Store domains are *your* input, so the policy gate cannot be settled once at
build time — every merchant writes their own `robots.txt`. Measured across
real stores:

| Store | `/products.json` |
| --- | --- |
| gymshark.com | ALLOWED |
| allbirds.com | ALLOWED |
| rothys.com | ALLOWED |
| fashionnova.com | ALLOWED |
| **shop.tesla.com** | **DISALLOWED** — `Disallow: /*.json` |

`shop.tesla.com` blocks exactly this endpoint. Worse, it does so with a
wildcard rule that **Python's own `urllib.robotparser` cannot see at all** —
it matches by prefix only, so it reads `Disallow: /*.json` as a rule about
paths literally starting `/*.json`, and would report `/products.json` as
ALLOWED. That's a false-*allow* at a policy gate: the dangerous direction.

This actor ships an RFC 9309 matcher, runs it per store before the first
product request, and turns a blocked store into an `ERROR` row naming the
rule. A store whose robots.txt gives AI crawlers a blanket `Disallow: /` is
skipped too.

### Prices without a currency are just numbers

`/products.json` carries `"price": "44.00"` and **no currency anywhere in the
payload** — no code, no symbol, no locale. Publishing that as a price is
publishing a number with an unknown unit, and Shopify stores serve different
currencies per region.

`/meta.json` has it, so this actor fetches it once per store (on by default)
and attaches the currency to every price-bearing row. It also yields the money
format, store name, `myshopify` domain, country — and the store's own
published-product count, which becomes a real completeness denominator.

That count is trustworthy, unusually: walking one store gave 38 pages × 250 +
97 = **9,597**, exactly matching its `published_products_count`.

**With `fetchStoreMeta` off, rows say `currency: null`.** They never guess USD.

### A collection's `products_count` over-counts what you can actually fetch

Measured on one store, comparing the claim against a full fetch — counting
only collections that returned fewer than 250 rows, so this cannot be a
pagination artifact:

| collection | claims | actually serves |
| --- | ---: | ---: |
| outlet-10 | 21 | **1** |
| 2-inch | 72 | **12** |
| 315 | 96 | **62** |
| 2-in-1-shorts | 150 | **69** |
| 3-inch | 398 | **115** |
| 4-inch | 462 | **148** |
| outlet-20 | 1 | **0** |

Seven of twelve sampled, every one over-counting — `products_count` includes
products not published to the online-store sales channel. It is published as
`collectionProductsCountClaimed`, never as a total, and
`countCollectionProducts` walks each collection for the real figure.

### Other things it handles

- **`limit` silently clamps at 250** on both endpoints. 251, 500 and 1000 all
  return a *byte-identical* 250-item response with no error. Capped in code.
- **Pagination ends honestly — and then 400s.** An empty page on an HTTP 200
  is the real end (it does *not* re-serve page 1). But past some depth
  Shopify answers **HTTP 400**, which means "no more", not "retry" — a generic
  retry ladder would spend four attempts discovering the end of a catalogue.
- **A 404 means "not a Shopify store", not "no products".** `bombas.com`
  answers 404 and serves `powered-by: Next.js`. Reported as a distinct error.
- **An unknown collection handle is an honest empty** (HTTP 200, `products: []`) — *not* the unfiltered catalogue. But that is indistinguishable from a
  genuinely empty collection, so the summary says `collectionReturnedNothing`
  rather than pretending to know which.
- **There is no `inventory_quantity` anywhere.** Stock is a boolean
  `available` per variant and nothing more. Stated as
  `inventoryQuantityIsNotAvailable` so a missing column reads as a limit of
  the endpoint, not a gap in this actor.
- **Order is stable and reverse-chronological** by `published_at` — verified
  by fetching the same page twice and getting identical id order — so a walk
  is repeatable and new products appear at the front.

### Output

One dataset, `recordType` tells the rows apart:

- `SEARCH_SUMMARY` — one per store: currency and store meta, the store's own
  published count, completeness ratio, the robots.txt verdict and matched
  rule, stop reason, duplicates dropped.
- `PRODUCT` — one per product, upstream's object verbatim plus normalised
  price range, variant/stock counts and image data.
- `VARIANT` — optional, one per SKU with price, compare-at price, sale flag,
  availability and option values.
- `COLLECTION` — one per collection, with claim and reality side by side.
- `ERROR` — one per store that failed, so every input maps to ≥1 row.

### Anti-bot and limits

**No WAF.** Seven TLS profiles returned 200 with byte-identical 39,119-byte
bodies on a store **fronted by Cloudflare** (`server: cloudflare`) — Cloudflare
in front of a store does not mean the storefront JSON is defended. A proxy is
optional and **off by default**.

**But some stores rate-limit datacentre IPs.** On the first production run,
`bombas.com` answered **HTTP 429** from an Apify datacentre IP where it
answers 404 from a residential one. Those two are never conflated here — 404
is a durable fact about the domain, 429 is a fact about the caller — so a
rate-limited store ends as `fetch_failed`, never as "not a Shopify store".
This is the reason the proxy option exists despite there being no WAF.

### Policy

`respectRobotsTxt` is on and cannot be turned off. Each store's own rules are
fetched and applied before any product request; an unavailable (4xx)
robots.txt is treated as "no restrictions" per RFC 9309 §2.3.1.3, **except**
when the 4xx body carries a WAF challenge — that means the store's real rules
are unknown, and the actor will not guess that they permit crawling.

Product data belongs to the merchants; check their terms before
redistributing it.

# Actor input Schema

## `mode` (type: `string`):

products = every product in the store (or in one collection). collections = the store's collection list, optionally with a real product count for each.

## `storeDomains` (type: `array`):

A domain (gymshark.com) or any URL on the store. Each store's own robots.txt is checked before a single product request is made — store domains are your input, and at least one real store (shop.tesla.com) forbids this exact endpoint with Disallow: /\*.json.

## `collectionHandle` (type: `string`):

products mode only. The handle from a collection URL, e.g. 'mens-shorts' from /collections/mens-shorts. Note Shopify answers an UNKNOWN handle with HTTP 200 and an empty list — identical to a genuinely empty collection — so the summary reports collectionReturnedNothing rather than guessing which it was.

## `maxProductsPerStore` (type: `integer`):

Set 0 for unlimited. Large stores are big — one measured store publishes 9,597 products across 39 pages. The summary reports completenessRatio against the store's own published count so you can see how much you got.

## `pageSize` (type: `integer`):

Between 1 and 250. Shopify silently clamps anything above 250 — 251, 500 and 1000 all return a byte-identical 250-item response with no error.

## `fetchStoreMeta` (type: `boolean`):

STRONGLY RECOMMENDED. /products.json contains NO CURRENCY anywhere — a variant says "price": "44.00" and nothing more. This costs one request per store and supplies the currency, money format, store name, myshopify domain and the store's own published-product count. With it off, every price is published with currency: null rather than a guess.

## `includeVariantRows` (type: `boolean`):

Emit a VARIANT row per SKU with its own price, compare-at price, availability and option values, alongside the PRODUCT rows. A store with 250 products can easily have 1,500 variants.

## `includeDescriptionHtml` (type: `boolean`):

body\_html is by far the largest field in the payload and most listings never use it, so it is dropped by default rather than silently truncated.

## `countCollectionProducts` (type: `boolean`):

collections mode only. Shopify's own products\_count OVER-COUNTS what the JSON endpoint will serve — measured claims of 21, 72, 96, 150, 398 and 462 against 1, 12, 62, 69, 115 and 148 actually served, because it counts products not published to the online store. This walks each collection to get the real figure. Costs one or more requests per collection.

## `respectRobotsTxt` (type: `boolean`):

Always on and cannot be disabled. Kept visible so the behaviour is stated rather than implied.

## `maxConcurrency` (type: `integer`):

Stores in flight at once. Pages within one store are sequential.

## `minRequestInterval` (type: `integer`):

Politeness pacing shared across all workers. 0 uses the built-in default.

## `proxyConfiguration` (type: `object`):

Optional and OFF by default. Storefront JSON is public and keyless and showed no WAF even on a store fronted by Cloudflare — all seven TLS profiles returned byte-identical responses.

## Actor input object example

```json
{
  "mode": "products",
  "storeDomains": [
    "allbirds.com",
    "https://www.fashionnova.com"
  ],
  "collectionHandle": "mens-shorts",
  "maxProductsPerStore": 250,
  "pageSize": 250,
  "fetchStoreMeta": true,
  "includeVariantRows": false,
  "includeDescriptionHtml": false,
  "countCollectionProducts": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 3,
  "minRequestInterval": 0,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "storeDomains": [
        "gymshark.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/shopify-products-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "storeDomains": ["gymshark.com"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/shopify-products-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "storeDomains": [
    "gymshark.com"
  ]
}' |
apify call scrapyx/shopify-products-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/shopify-products-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/esvq3xHkqhlFi2bsf/builds/ShiYL67A3NPCYUtGN/openapi.json