# Changelog of Shopify Store Leads — Catalog Size, Apps, Contacts (`autofacts/shopify-store-leads`) Actor - **URL**: https://apify.com/autofacts/shopify-store-leads/changelog.md - **Full Actor documentation**: https://apify.com/autofacts/shopify-store-leads.md ## Changelog ### Unreleased #### Added - `useShopApp` — the Shop app, Shopify's marketplace, as a source. On by default; `false` turns it off. With a keyword, or the product category when no keyword is set, its product search supplies the candidate stores, and nothing else does: about twenty a request, each already selling what was asked for, where the seed hands out stores in file order whatever the keyword. The seed and Common Crawl serve only runs with neither. A store found this way counts as matching that keyword; every other filter still applies. Every store that passes the filters is then found on the Shop app at a new funnel stage 4 — by that search, or by a lookup of its own name matched on its Shopify id — for its rating, its business address and the contacts it lists there, and the store's own endpoints fill in the rest. The Shop terms forbid automated access and getting past access measures (research/shop-app-2026-09-23.md §5); on 2026-09-23 the owner chose to use it anyway, first as an opt-in source and then, after a live run the same day, by default. That run, local through Apify Proxy, keyword "coffee", `maxItems: 30`: 30 stores stored from 36 examined in about six minutes, all found by the search; one Shop app refusal in 33 requests, which the retry got past. Its pace was set by Shopify's `/meta.json` limit, not the Shop app's. The name lookup, run live once on `gorjana.com` from `storeUrls`, found the store by its Shopify id and brought back its rating, a three-line address, and a phone number and a Pinterest profile that its storefront does not link. - `rating` — the store's Shop app rating: the average, the ratings behind it and the written reviews. Null when the Shop app is off, does not list the store under its name, or has no ratings for it. No request of its own: it comes with the search result or the lookup. 30 of 30 stores carried one in the live run. - `address` — the business address a store gives the Shop app, one entry per line as the Shop app formats it, verbatim. Read from the store's Shop app page once the store has passed every filter but `requireEmail` and `requirePhone`, when contacts are on — one request at stage 4, which the old stage 4, the store's own contact page, now follows as stage 5. The page's email, phone and social profiles come first in the store's contacts, and the storefront and the contact page fill in the rest, all through the same email test; a store whose page gives an email no longer needs its contact page fetched. The page is used only when it names the store's own myshopify domain. 25 of 30 stores carried one in the live run. #### Changed - Every email address a store publishes is kept, not only role addresses. A list of role words — `info@`, `support@` and twenty more — dropped every other address as a person's, which cost the brand inboxes the list did not know: of the 12 emails clearpath returned and this actor did not on the same 98 stores, 8 were `hey@`, `cs@`, `concierge@` and the like, on pages the actor had read. On 2026-09-24 the owner decided that an address a store publishes is a business contact, whoever's name is in it. What stays out is what nobody can write to: `no-reply@` and its kin, the placeholders themes and apps ship with, image names like `logo@2x.png`, and runs of characters longer than an address can be. Two fragments the role test used to hide are now cut where they belong: a JSON escape before an address in a script (`\u003e`, `\n`), and a word a link splits (`support@`). On 98 saved homepages, the homepage alone gives an email for 45 stores, against 30 before. #### Fixed - A store Shopify refuses is no longer dropped, and no longer slows the run. Through the proxy each store's requests leave from one IP, and a 429 is that IP's, but the storefront client retried it from the same IP and slowed every other request as if the limit were the run's. On the platform on 2026-09-24 a 30-store keyword run slowed eleven times, seven of them to one request per 5 s, dropped five stores after three 429s each, and took 359 s; those five stores then answered at the first try from three other IPs each. A failed request now goes out again on a new proxy session, which its store keeps, and nothing else waits. The same run locally through Apify Proxy: three 429s, each answered from the next IP, no store dropped, 30 in 108 s. On the platform on 2026-09-25, under a tighter limit — a third of first `/meta.json` requests refused — clearpath's 100 domains stored 98 in 422 s, against 85 in 1,413 s the day before: 32 of the 35 refused requests were answered from another IP. Without a proxy, where every request shares one IP, a 429 still slows the whole run. - A keyword search no longer stalls after its first page. Every Shop app request went out from one IP: an impit instance keeps its connection to a host open, and a session-less proxy URL changes the IP only per connection. The Shop app refuses an IP after a search page and a few store pages, so on the platform two coffee runs lost page 2 and both retries of it, fell back to walking the seed — one store every seven to eight minutes — and one of them timed out at the hour with 26 of 30. Each Shop app request now goes out from a new instance on a new proxy session, and a refused one is sent again from another IP, five times for a search page and three for a lookup or a store page, half a second apart instead of two. Locally through Apify Proxy, "coffee" with `maxItems: 30`: 30 stored in 97 s, page 2 at the first try. When the search itself is refused or runs out, the run ends and says so, with the stores it has. - The actor runs in 256 MB. It parsed each store's homepage whole and kept the parsed page while that store waited on its later requests, and a parsed homepage takes 20–30 times the memory of its HTML: one coffee roaster's 5.7 MB homepage, most of it inline SVG, came to 162 MB. On 2026-09-23 a 100-store run reached the platform's 512 MB limit, and in a local container capped at 256 MB the previous build was killed six seconds in. Each page is now read once, as it arrives, without the scripts, styles and SVG images that no field is read from, and only what the row needs is kept. In the same container, the same 100 stores: all stored, peak 112 MB. On 102 saved homepages every field comes out as it did before. - A restarted run no longer stores and charges again the stores it had already stored, and `maxItems` caps the run rather than each attempt at it. The platform restarts a run that exits non-zero with the same default dataset but a new process, and the set of stores already charged and the count against `maxItems` lived only in memory. Measured on 2026-09-23: a run given 100 domains and `maxItems: 100` exited 83 s in, very likely out of memory, and the restart started the input over — 112 rows and 112 charges for 99 stores. The crawler now reads its own default dataset back on start, on the platform only, and restores both. - Runs no longer report to Scraper Insight under shopify-scraper's identity. This actor was forked from that one and carried its app id and secret verbatim, so every store-discovery run was filed against shopify-scraper and moved its request count, failure rate and distinct-account count. The actor is now registered as its own app and carries that pair. ### 0.2 — 2026-09-16 #### Fixed - The prefilled store list now exercises every field. It was `gorjana.com`, `allbirds.com` and `kith.com` — three large brands that between them returned `announcementBarText: null` and `policies: []` on all three rows, so the sample output a buyer sees first showed both fields permanently empty. It is now `gorjana.com` (US, 976 products, emails and apps), `tentree.com` (US, 6,498 products, an English announcement bar) and `ayvii.com` (AT, 13 products, five policies and a German bar). That also drops the slowest store: `kith.com` carries 25,001 products and took most of the daily test's runtime. - The README no longer claims a rotating bar returns only its first message. Whether it does depends on the theme: `tentree.com` puts every slide in one element and returns all four promotions joined. The new prefill surfaced this, and the field's behaviour is documented as it is rather than as intended. - The README no longer implies that eight in ten stores of every kind publish canonical policy URLs. Measured across nine well-known direct-to-consumer brands, one did. - The default proxy no longer names a group. It was `["DATACENTER"]`, which is validated against the *caller's* account — and on current Apify plans the shared datacenter pool is a paid add-on, so the default input failed validation before the run could start: *"You currently do not have access to proxy groups: DATACENTER"*. That broke the actor for every buyer without the add-on, and for the daily auto-tester, which runs the prefilled default. `{"useApifyProxy": true}` lets each run use whatever groups its own account has. #### Added - `announcementBarText` — the storefront's announcement bar message, verbatim and in the store's own language, or `null`. No extra request: the bar is in the homepage already fetched for the theme and apps. Measured non-null on 18 of 29 live storefronts. Deliberately not parsed into a free-shipping threshold, a discount percentage or a promo code — the copy is in the merchant's own language, and a percent-off pattern run over a storefront matches product prices rather than promotions. #### Fixed - `activity.productsPerMonth` and `policies` appear in the `overview` dataset view. Both shipped on the record on 2026-09-15 with display labels already written, but neither was added to the view's field list, so neither has ever rendered in the Store's default view. - `policies` declared in `dataset_schema.json`. It has shipped on the record since 0.1 but was missing from the field list, so the Apify Store's field table omitted it. ### 0.1 — 2026-09-15 First version. Finds and qualifies Shopify stores, one billable row per store. #### Added - Enrichment path: `storeUrls` in, full `Store` record out. - Discovery path: a bundled seed of 6,542 verified stores, then the Common Crawl URL index (`ccIndex`, newest crawl resolved at run time when unset). Candidates are drawn lazily — index pages are fetched only once the cheaper sources run out — and a run that reaches its scan budget before filling `maxItems` says so rather than returning quietly short. - Filters — `countries`, `query`, `excludeKeywords`, `minProducts`, `maxProducts`, `category`, `activeWithinDays`, `shipsToCountries`, `minAvgPrice`, `maxAvgPrice`, `inStockOnly`, `hasApps`, `appMatchMode`, `excludeApps`, `hasSocials`, `requireEmail`, `requirePhone` — each evaluated at the earliest funnel stage whose data can decide it, so a rejected store costs as little as one request, and none of them adds a request to a run. - `Store` record: identity, catalog (product and collection counts, price band, product types, vendors, size band), activity (newest and oldest product, days since last product, `isActive`, new products per month), published policy pages, tech stack (theme, installed apps, card brands) and business contacts. - Rate-limit handling: `Retry-After` is honoured, a 429 slows every request rather than being retried straight into the same limit, and a run refused ten times in a row stops instead of spending the rest of its budget being blocked. The proxy defaults to Apify Proxy's datacenter group, because the limit is per client IP and rotation is the answer to it. - Contact extraction limited to business role addresses, with Cloudflare email-protection decoding and same-domain script-blob extraction. The store's contact page is fetched only when the homepage published no address, and only by following the store's own link. - `previousDatasetId` accepted and ignored, reserving the name so change tracking can be added later without a breaking input change. - `dataset_schema.json` with `views` — `overview` and `catalog` — so the Apify Store renders a field table. - 62 tests: `test/funnel.test.mjs` pins the exact request count for every rejection path, `test/enrich.test.mjs` covers the extraction functions, and `test/candidates.test.mjs` pins when a Common Crawl index page is fetched.