Shopify Store Leads Scraper

Apify Actor (v1). Given a list of Shopify store domains, fetches each store's public product catalog via the standard, unauthenticated products.json endpoint and normalizes it into a lead-gen dataset: one summary row per store (catalog size, price range, vendors carried, activity signal) plus one row per product. Pay-per-event pricing, charged per product row.

Project structure

.actor/
├── actor.json          # Actor metadata, memory limits
├── input_schema.json   # Input form: storeDomains, includeProductRows, maxProductsPerStore
└── dataset_schema.json # Dataset overview table (Console UI)
src/
├── main.js             # Actor entry point: init, orchestrate, charge, exit
├── shopify.js           # Network layer: products.json pagination, store meta fetch, .myshopify.com Hydrogen/Oxygen fallback (plain fetch(), no browser)
├── normalize.js          # Pure raw-Shopify-JSON -> unified-schema mapping (unit tested)
├── orchestrate.js        # Pure orchestration logic (unit tested), wired to Actor + shopify.js in main.js
└── validateInput.js      # Pure input-completeness / domain-list cleanup (unit tested)
test/                    # Vitest suite (no live network calls)
Dockerfile               # apify/actor-node:20 base image - no Playwright/Chrome needed

Why no Playwright / Crawlee browser automation

{store}/products.json is Shopify's standard, undocumented-but-widely-known public storefront API - plain JSON, no auth, no JS rendering required. Verified directly with curl and confirmed end-to-end with this Actor's own code against real stores (see "Verification against real stores" below), so this Actor uses plain fetch() with manual retry/backoff, not a headless browser. That keeps the Docker image small (apify/actor-node:20, no Chrome) and the run cheap/fast, which matters directly for PPE margin.

Output schema

Two row shapes share one dataset, distinguished by recordType:

recordType: "store" - one per input domain, always pushed (even for stores that failed, so failures are visible in the dataset, not just logs):

{
  "recordType": "store",
  "domain": "taylorstitch.com",
  "storeUrl": "https://taylorstitch.com",
  "storeName": "Classic Men's Clothing | Taylor Stitch",
  "storeDescription": "...",
  "productCount": 30,
  "vendorCount": 1,
  "vendors": ["Taylor Stitch"],
  "productTypeCount": 5,
  "productTypes": ["Knits", "Wovens", "..."],
  "priceMin": 55,
  "priceMax": 248,
  "oldestProductCreatedAt": "2025-12-12T19:21:16.000Z",
  "mostRecentProductUpdatedAt": "2026-08-09T07:56:03.000Z",
  "appearsActive": true,
  "pagesFetched": 1,
  "error": null,
  "usedMyshopifyFallback": false,
  "scrapedAt": "2026-08-09T07:56:02.995Z"
}

appearsActive is a heuristic: true if any product was updated in the last 90 days, null if there's no product data to judge from (e.g. the store fetch failed). error is null on success, or a human-readable reason (HTTP status, non-JSON response, etc.) on failure - one store failing never stops the others (see runShopifyScrape in src/orchestrate.js). usedMyshopifyFallback is true when the custom domain's own /products.json failed and this Actor recovered the store's real catalog from its underlying {handle}.myshopify.com domain instead (see "Headless Shopify Hydrogen/Oxygen fallback" below); domain/storeUrl still reflect the original customer-facing domain either way.

recordType: "product" - one per product, pushed only if includeProductRows is true (default) and the store fetch succeeded:

{
  "recordType": "product",
  "domain": "taylorstitch.com",
  "storeUrl": "https://taylorstitch.com",
  "productId": 7645559554125,
  "title": "The Regenerative Cotton Tee in Saffron",
  "handle": "regenerative-cotton-tee-in-saffron-2608",
  "productUrl": "https://taylorstitch.com/products/regenerative-cotton-tee-in-saffron-2608",
  "vendor": "Taylor Stitch",
  "productType": "Knits",
  "tags": ["BASICS", "KNITS", "..."],
  "priceMin": 55,
  "priceMax": 55,
  "variantCount": 6,
  "images": ["https://cdn.shopify.com/..."],
  "createdAt": "2025-12-12T19:23:10.000Z",
  "updatedAt": "2026-08-09T07:56:03.000Z",
  "publishedAt": "2026-08-07T16:48:58.000Z",
  "scrapedAt": "2026-08-09T07:56:02.995Z"
}

Known limitation: priceMin/priceMax have no currency field - Shopify's products.json doesn't include one (currency is set at the store/checkout level, not per-product). Buyers need to infer currency from the store's market themselves; this Actor doesn't guess it.

Input

{
  "storeDomains": ["allbirds.com", "gymshark.com"],
  "includeProductRows": true,
  "maxProductsPerStore": 5000
}

• storeDomains (required) - list of domains or full URLs. Each is fetched independently; one bad domain doesn't stop the others.
• includeProductRows (default true) - set false to only get store summary rows (cheaper: no product-event charges, just store rows).
• maxProductsPerStore (default 5000, max 100000) - safety cap on pagination per store, protects against runaway cost on huge catalogs.

Running locally

npm install
mkdir -p storage/key_value_stores/default
cat > storage/key_value_stores/default/INPUT.json << 'EOF'
{ "storeDomains": ["allbirds.com", "gymshark.com"], "includeProductRows": true, "maxProductsPerStore": 100 }
EOF
APIFY_LOCAL_STORAGE_DIR=$(pwd)/storage node src/main.js

Dataset rows land in storage/datasets/default/; run status/warnings print to the console log.

Tests

$npm test

49 tests across 5 files, all fixture/mock-based (no live network calls in the automated suite, per spec):

• test/normalize.test.js - raw Shopify product -> normalized product row, and store-level aggregation (vendors, product types, price range, appearsActive heuristic, empty-catalog/error case).
• test/shopify.test.js - pagination stopping conditions (empty page, partial page, maxProducts cap mid-page), 429 retry-with-backoff, network error retry-then-fail-fast (no wasted sleep on the last attempt), normalizeStoreInput's handling of an explicit port (kept in baseUrl, stripped from the display-only domain), error handling for non-JSON/blocked responses and non-retryable 4xx, extractMyshopifyHandle/ isHydrogenOxygenPoweredBy unit tests, and fetchStoreData's .myshopify.com fallback path (rescue on 404, rescue on Hydrogen/Oxygen redirect, no-handle-found fails closed, handle-found-but-also-fails falls back to the original error, no self-referential retry loop, and no fallback attempted when the primary fetch already succeeds) - global.fetch is mocked, no real network calls.
• test/orchestrate.test.js - per-store failure isolation (one store erroring doesn't stop or corrupt others), includeProductRows toggle, and spending-limit (eventChargeLimitReached) stop-early behavior.
• test/validateInput.test.js, test/input-schema.test.js - input validation and domain-list cleanup (dedup, trim, drop empties).

Verification against real stores (honesty report)

Premise check, before building (per the brief, since the sibling project apify-review-intel was paused after building on an unverified bot-protection assumption): before writing any Actor code, I curl'd /products.json on 12 real Shopify-platform stores beyond the two originally given (allbirds.com/gymshark.com), with a browser User-Agent, no proxy:

9 of 12 (75%) plain, unauthenticated, working products.json - premise holds broadly, with real exceptions that are worth building for rather than ignoring: (a) some domains that sound like Shopify stores aren't Shopify at all, (b) a small number of stores do run bot-mitigation (here, Vercel) in front of or instead of raw Shopify. Both are handled by returning a descriptive per-store error field instead of crashing or silently dropping the store.

End-to-end runs of the actual built Actor (not just curl) against fresh real stores, none of them allbirds.com/gymshark.com:

Run 1 - colourpop.com, taylorstitch.com, gfuel.com, bombas.com, mvmt.com (30 products/store cap): 3/5 succeeded (colourpop, taylorstitch, gfuel - 30 real products each, real prices/vendors/tags/timestamps in the output). bombas.com and mvmt.com failed with the errors above, each still producing a recordType: "store" row with error populated and productCount: 0 - no crash, no silent drop, and (importantly for PPE billing) zero product events charged for the two that failed, since product rows are only pushed for products that were actually fetched.

Run 2 -

kithnyc.com, hauslabs.com, steepandcheap.com, allplants.com, fashionnova.com

Non-JSON response ... likely not a Shopify store, or blocked by a bot-detection/challenge page

Total across both runs: 7 of 10 fresh real stores succeeded (colourpop, taylorstitch, gfuel, kithnyc, hauslabs, allplants, fashionnova), all with real product data (titles, prices, vendors, tags, timestamps) verified by eye in the output dataset. 3 failed for real, distinct, explainable reasons (one sunset/non-Shopify domain, one non-Shopify platform, one genuine bot-mitigation vendor) - none of which are code bugs, and all of which are surfaced to the user as a per-store error field rather than silently missing data or crashing the whole run.

Independent tester sample (larger, more honest number): the above 10 domains were curated by me while building the Actor, which likely biased the sample toward stores that were already known/expected to work. An independent tester later ran a fresh, separately-chosen sample of 21 additional Shopify-adjacent domains and found 13/21 (~62%) succeeded with real product data, and 8/21 failed - in three distinct, now-confirmed categories:

1. Genuine bot-mitigation (Cloudflare challenge pages, e.g. thefarmersdog.com, hydroflask.com) - same category as bombas.com above.
2. Not actually Shopify, despite the domain sounding like a plausible target (welly.com, purple.com, moment.com, bellroy.com) - same category as mvmt.com above.
3. Headless Shopify Hydrogen/Oxygen storefronts (ruggable.com, nomadgoods.com) - a category this Actor's own pre-build sample didn't surface. These are real, live Shopify-cdn stores, but they run a custom Hydrogen/Oxygen (React/Next.js-style) storefront in front of Shopify, so the classic /products.json endpoint at the root domain either 404s or redirects instead of resolving the normal way - confirmed via the powered-by: Shopify, Oxygen, Hydrogen response header and custom Next.js-style routing on those domains. Not a bot block and not "not-Shopify" - just a storefront architecture this Actor's plain /products.json approach doesn't handle. The powered-by header is a plausible future detection signal (to at least report a more specific error, e.g. "headless Hydrogen storefront, products.json not at expected path" instead of a generic non-JSON error) - noted here as a documented pattern, not built in v1.

Combined honest number (pre-fix baseline): across my original 10 and the tester's independent 21 (31 distinct real domains total, no overlap), overall observed success rate was ~65% (20/31), not the ~90% the smaller, pre-build curated sample implied. That was the number to plan around for realistic yield when running this Actor against a general list of "looks-like-Shopify" domains - roughly a third of any such list was expected to fail for one of the three reasons above.

v1.1: .myshopify.com fallback for headless Hydrogen/Oxygen storefronts (fixes category 3)

Hypothesis tested before writing any code (per the brief): a merchant running a headless Hydrogen/Oxygen frontend on their custom domain typically still has their underlying {handle}.myshopify.com domain active on Shopify's own platform, even though the custom domain's /products.json doesn't resolve normally - so that .myshopify.com domain's /products.json might still work.

Verified manually against both real, named category-3 examples before building anything:

The hypothesis held for both real examples - implemented.

Important honest correction to the original plan: the spec's proposed detection gate - trigger the fallback only when the

powered-by: Shopify, Oxygen, Hydrogen

What was actually built instead (src/shopify.js): when the primary domain's /products.json fails for any reason, and the store's own homepage HTML contains exactly one unambiguous {handle}.myshopify.com reference (extractMyshopifyHandle), the Actor retries /products.json against https://{handle}.myshopify.com before giving up. This is deliberately conservative, not a blind guess:

• If zero or more-than-one distinct .myshopify.com handles are found on the homepage, no fallback is attempted at all - ambiguity fails closed into the original error rather than guessing which one might be right.
• The fallback fetch is validated exactly like the primary fetch (real HTTP request, real {"products": [...]} shape check, same retry/backoff) - if it also fails, the original error is what gets reported, never fabricated or wrong data.
• The powered-by: Shopify, Oxygen, Hydrogen header is still checked and, if present, appended as a hint to the error message on failure (useful diagnostic context) - but it is no longer the trigger for whether the fallback is attempted, since that would have missed ruggable.com.
• Output domain/storeUrl always stay the original customer-facing domain the user asked for; only the underlying network fetch target changes. A new usedMyshopifyFallback boolean field on the store row (see "Output schema" above) makes rescued rows visible without needing to check logs.
• Existing product/store PPE charge events, input schema, and the primary fetch path for categories 1 and 2 (not-actually-Shopify, bot-mitigation) are untouched.

False-positive check: re-tested the six other real category-1/2 failure domains named above (welly.com, purple.com, moment.com, bellroy.com, thefarmersdog.com, hydroflask.com) plus mvmt.com, bombas.com, steepandcheap.com

• none of their homepages contain any .myshopify.com reference, so the fallback correctly never fires for genuinely non-Shopify or genuinely-bot-blocked domains; they still fail with their original, accurate error messages.

Regression + rescue re-test (npm test: 49/49, up from 36/36 - 13 new mock-based tests for extractMyshopifyHandle, isHydrogenOxygenPoweredBy, and the fetchStoreData fallback path, no live network in the automated suite; plus a live re-run of the actual built Actor against 20 real domains - the original 10 known-good curated stores, the 8 named tester failures, plus allbirds.com/gymshark.com):

Result on this 20-domain sample: 9/20 → 11/20 (45% → 55%), with zero regressions on any of the other 18 domains.

Updated combined honest number: the README's original 31-domain combined sample named exactly two category-3 (headless Hydrogen/Oxygen) failures - ruggable.com and nomadgoods.com - and no others; the other 9 of 11 original failures were confirmed-unaffected categories 1/2 (bot-mitigation, not-actually-Shopify), re-verified above as still failing identically after this change. So the extrapolated updated combined success rate on that same 31-domain sample is ~71% (22/31), up from ~65% (20/31). This is an extrapolation, not a fresh independent re-run of all 31 named domains (the 13 domains the independent tester reported as succeeding were never individually named in this README), but it's a defensible one: this fix is additive and regression-free (verified above), so it can only ever move domains from failing to succeeding, never the reverse, and the only two category-3 domains on record are both now rescued.

Known residual limitation: this fallback only works when the merchant's headless frontend still loads something on the homepage that references a .myshopify.com handle (an analytics script, A/B-testing config, etc.) - both real examples tested had this. There are two distinct failure modes, not one:

1. Fails closed (safe): the homepage has zero such references, or two or more distinct ones (e.g. a third-party embed referencing an unrelated store's handle alongside the real one). Both cases are treated as ambiguous/not-found and the Actor returns the original error - no wrong data.
2. Does not fail closed (real, unmitigated risk): the homepage has exactly one .myshopify.com reference, but it happens to belong to an unrelated store (e.g. a "curated by our partner" widget with no genuine reference to the requested store's own handle anywhere on the page) rather than the requested store's own backing shop. There is no ownership-verification step - the code only checks "is there exactly one candidate," not "is this candidate actually this store's own shop." In that case the fallback returns the unrelated store's real product data, mislabeled under the requested domain, with error: null and usedMyshopifyFallback: true - indistinguishable from a correct rescue without manually checking the myshopifyDomain field against the expected brand. Confirmed exploitable in adversarial testing; not observed on any of the 20+ real domains tested so far, but not structurally prevented either. Treat myshopifyDomain on any row with usedMyshopifyFallback: true as worth a manual glance before trusting the catalog data at face value. No paid third-party API/service was used or is required for this fallback.

What NOT built in v1 (out of scope, as specified)

• No store-discovery/crawling feature - input is an explicit domain list.
• No email/contact enrichment.
• No scheduling/monitoring/diffing between runs.
• No competitor-comparison features.
• No currency detection/normalization for prices (see "Known limitation" above).
• No handling for Shopify stores that gate products.json behind a password-protected storefront (password-protected dev/staging stores return HTML, not JSON, for every endpoint including products.json - this Actor will report those as a non-JSON error, same as any other non-standard-setup store; not specifically tested since none of the sampled real stores were password-protected).
• No special-case detection/handling for headless Shopify Hydrogen/Oxygen storefronts - built in v1.1, see ".myshopify.com fallback for headless Hydrogen/Oxygen storefronts" above. Still out of scope: rescuing a headless storefront whose homepage doesn't embed any discoverable .myshopify.com reference at all, or store-discovery beyond what's reachable from the homepage HTML already fetched for store metadata.

Pay-per-event (PPE) pricing - what's implemented vs. what needs Console setup

Per the same Apify PPE convention used in the sibling project, the actual dollar price per event is configured in Apify Console, not in this repo. What this Actor does on the code side:

• Every product row is pushed via Actor.pushData(record, 'product') - one charge per product, matching the research's target pricing framing ("$0.50-3/1,000 products scraped").
• Every store summary row (including failed stores, so failures are visible) is pushed via Actor.pushData(summary, 'store') - a separate custom event from product, so per-store overhead can be priced independently (or left at $0) without double-counting against the per-product price. This was a judgment call: the spec says "charge a custom event (e.g. product) once per Actor.pushData(item, 'product') call" for the product metric specifically, and doesn't specify pricing for the summary row, so a distinct store event keeps the two concerns billable independently rather than either force-fitting store rows into the product event (over-charging: a store row isn't a product) or leaving store rows unbilled with no event at all (under-specified). Flagging this choice explicitly in case Coordinator/pricing owner wants it done differently.
• The code respects ChargeResult.eventChargeLimitReached and stops pushing further product rows (and further stores) once the user's run spending limit is hit.
• minMemoryMbytes/maxMemoryMbytes are set low (256-1024 MB) in .actor/actor.json since there's no browser to run.

Needs to be done in Apify Console before publishing (not possible from code):

1. Enable pay-per-event monetization on the Actor.
2. Add and price the product custom event (target ~$0.50-3/1,000, per research).
3. Add and price the store custom event (per-store summary row overhead; consider $0 if the model should be purely per-product).
4. Reconcile with apify-default-dataset-item (Apify's synthetic per-dataset-item event) the same way the sibling project flagged - since this Actor uses custom product/store events on every pushData() call, leaving apify-default-dataset-item enabled by default would double-charge. Must be explicitly disabled/reconciled in Console.

Other things flagged as uncertain / needing verification

• Apify CLI: not installed/exercised in this sandbox - only the direct APIFY_LOCAL_STORAGE_DIR=... node src/main.js invocation was tested. The Tester should confirm apify run behaves the same.
• Apify Proxy: all testing above ran with the sandbox's plain outbound network, no Apify Proxy. Since /products.json needed no proxy in every test here, this Actor does not call Actor.createProxyConfiguration() at all - if a wider production sample turns up more stores like bombas.com (real bot-mitigation), adding an optional proxy pass-through would be the natural v1.1 follow-up, not something built speculatively now.
• Broader sample size: 43 total real domains were checked across the premise check, the two end-to-end runs, and the independent tester's larger follow-up sample (12 by curl, 10 by running the actual Actor, 21 by the independent tester, some curl/Actor overlap; 31 distinct domains counted toward the combined ~65% success rate above). That's a more credible sample than the original curated one, but still not exhaustive - true success rate at full Shopify-store-population scale could differ further from the ~65% observed here, and the three named failure categories (bot-mitigation, not-actually-Shopify, headless Hydrogen/Oxygen) may not be exhaustive either.