Shopify Products Scraper

Products, variants and collections from any Shopify store, via the public storefront JSON every store exposes on its own domain. No login, no API key, no app install, no browser.

It checks each store's robots.txt before it fetches a single product

Store domains are your input, so the policy gate cannot be settled once at build time — every merchant writes their own robots.txt. Measured across real stores:

Store	/products.json
gymshark.com	ALLOWED
allbirds.com	ALLOWED
rothys.com	ALLOWED
fashionnova.com	ALLOWED
shop.tesla.com	DISALLOWED — Disallow: /*.json

shop.tesla.com blocks exactly this endpoint. Worse, it does so with a wildcard rule that Python's own urllib.robotparser cannot see at all — it matches by prefix only, so it reads Disallow: /*.json as a rule about paths literally starting /*.json, and would report /products.json as ALLOWED. That's a false-allow at a policy gate: the dangerous direction.

This actor ships an RFC 9309 matcher, runs it per store before the first product request, and turns a blocked store into an ERROR row naming the rule. A store whose robots.txt gives AI crawlers a blanket Disallow: / is skipped too.

Prices without a currency are just numbers

/products.json carries "price": "44.00" and no currency anywhere in the payload — no code, no symbol, no locale. Publishing that as a price is publishing a number with an unknown unit, and Shopify stores serve different currencies per region.

/meta.json has it, so this actor fetches it once per store (on by default) and attaches the currency to every price-bearing row. It also yields the money format, store name, myshopify domain, country — and the store's own published-product count, which becomes a real completeness denominator.

That count is trustworthy, unusually: walking one store gave 38 pages × 250 + 97 = 9,597, exactly matching its published_products_count.

With fetchStoreMeta off, rows say currency: null. They never guess USD.

A collection's products_count over-counts what you can actually fetch

Measured on one store, comparing the claim against a full fetch — counting only collections that returned fewer than 250 rows, so this cannot be a pagination artifact:

collection	claims	actually serves
outlet-10	21	1
2-inch	72	12
315	96	62
2-in-1-shorts	150	69
3-inch	398	115
4-inch	462	148
outlet-20	1	0

Seven of twelve sampled, every one over-counting — products_count includes products not published to the online-store sales channel. It is published as collectionProductsCountClaimed, never as a total, and countCollectionProducts walks each collection for the real figure.

Other things it handles

• limit silently clamps at 250 on both endpoints. 251, 500 and 1000 all return a byte-identical 250-item response with no error. Capped in code.
• Pagination ends honestly — and then 400s. An empty page on an HTTP 200 is the real end (it does not re-serve page 1). But past some depth Shopify answers HTTP 400, which means "no more", not "retry" — a generic retry ladder would spend four attempts discovering the end of a catalogue.
• A 404 means "not a Shopify store", not "no products". bombas.com answers 404 and serves powered-by: Next.js. Reported as a distinct error.
• An unknown collection handle is an honest empty (HTTP 200,

  products: []
  ) — not the unfiltered catalogue. But that is indistinguishable from a genuinely empty collection, so the summary says collectionReturnedNothing rather than pretending to know which.
• There is no inventory_quantity anywhere. Stock is a boolean available per variant and nothing more. Stated as inventoryQuantityIsNotAvailable so a missing column reads as a limit of the endpoint, not a gap in this actor.
• Order is stable and reverse-chronological by published_at — verified by fetching the same page twice and getting identical id order — so a walk is repeatable and new products appear at the front.

Output

One dataset, recordType tells the rows apart:

• SEARCH_SUMMARY — one per store: currency and store meta, the store's own published count, completeness ratio, the robots.txt verdict and matched rule, stop reason, duplicates dropped.
• PRODUCT — one per product, upstream's object verbatim plus normalised price range, variant/stock counts and image data.
• VARIANT — optional, one per SKU with price, compare-at price, sale flag, availability and option values.
• COLLECTION — one per collection, with claim and reality side by side.
• ERROR — one per store that failed, so every input maps to ≥1 row.

Anti-bot and limits

No WAF. Seven TLS profiles returned 200 with byte-identical 39,119-byte bodies on a store fronted by Cloudflare (server: cloudflare) — Cloudflare in front of a store does not mean the storefront JSON is defended. A proxy is optional and off by default.

But some stores rate-limit datacentre IPs. On the first production run, bombas.com answered HTTP 429 from an Apify datacentre IP where it answers 404 from a residential one. Those two are never conflated here — 404 is a durable fact about the domain, 429 is a fact about the caller — so a rate-limited store ends as fetch_failed, never as "not a Shopify store". This is the reason the proxy option exists despite there being no WAF.

Policy

respectRobotsTxt is on and cannot be turned off. Each store's own rules are fetched and applied before any product request; an unavailable (4xx) robots.txt is treated as "no restrictions" per RFC 9309 §2.3.1.3, except when the 4xx body carries a WAF challenge — that means the store's real rules are unknown, and the actor will not guess that they permit crawling.

Product data belongs to the merchants; check their terms before redistributing it.