Crawl Shopify App Store categories and get per-app listings plus niche-opportunity metrics (reviews per app, top-3 concentration, share of apps under 100 reviews).
All notable changes to the Shopify App Store Analyzer are documented here.
This project follows Semantic Versioning .
[1.0.4] - 2026-08-19
Build pipeline: source now lives at github.com/xueleo000/gaplens-shopify-app-store-analyzer; every push to main runs CI (typecheck/test/build) and triggers an Apify build via webhook. No more manual apify push.
Publishing prerequisites and the first round of live-site fixes. The Apify Actor
version stays 1.0 in .actor/actor.json because the platform only accepts
MAJOR.MINOR there; 1.0.2 is the source/package version.
Added
.actor/output_schema.json (actorOutputSchemaVersion: 1), referenced
from actor.json as "output". This is what the Apify Console asks for with
"Add 'Output' schema(s) to the source code and rebuild the Actor before
publishing." It describes all four result destinations: the default dataset of
app records (raw and through the overview view), the OUTPUT record of the
default key-value store (category summaries, keyword clusters, run stats), the
named category-summaries dataset, and the DEBUG_* HTML snapshots.
Field-level schema for the default dataset in .actor/dataset_schema.json
(fields, JSON Schema draft-07). Deliberately permissive — nullable types, no
required, additionalProperties: true — so it documents items without ever
rejecting a push. The existing overview view is unchanged.
Description recovery for the data-attributes strategy
(extractCardDescription): when Shopify does not mirror the tagline into a
data-app-card-*-value attribute, the description is read from the card's
description element — a self-declared description/tagline/subtitle/line-clamp
node, a block following the title link, or the first non-rating, non-price text
block longer than 20 characters.
Cross-strategy description merge (mergeCardDescriptions): if the winning
strategy returns cards but more than half of them have no description,
descriptions are back-filled from the other strategies (heuristic first),
matched on handle, then canonical URL, then name. Only description is
touched.
9 new parser tests and two fixtures
(category-data-attributes-sibling-desc.html,
category-data-attributes-merge.html) covering in-card description elements,
the merge path and the reject rules (rating text, price labels, name+tagline
wrappers).
tests/actor-definition.test.ts (13 cases) — guards the files the platform
reads: the schema references in actor.json, the MAJOR.MINOR version format,
a changelog entry per package version, title/template on every output
property, the overview view's fields all being declared in fields, permissive
dataset validation, enablePpeCharging defaulting to false, and no price
literal anywhere in src/. Suite is now 67 tests.
Fixed
description was null on every live record. The live smoke test picked
data-attributes (24 cards) over heuristic (26 cards with descriptions), and
the data-attribute path only ever read the description from an attribute that
the live pages do not emit. Both changes above close that gap.
Changed
Monetization docs now match the Console. Pricing is Apify's
platform-managed pay-per-event events — apify-actor-start ($0.05) and
apify-default-dataset-item ($0.005) — which the platform meters itself. The
README pricing table, .actor/pay_per_event.json and the enablePpeCharging
input description were rewritten accordingly.
Custom charge events stay off by default.enablePpeCharging now defaults
to false in input_schema.json, and src/main.ts no longer turns charging on
automatically when ACTOR_MAX_TOTAL_CHARGE_USD is present in the environment —
that auto-detection would have double-billed every run, since the custom
app-record event duplicates the platform-managed apify-default-dataset-item
charge. The Actor.charge code path is kept for a possible switch to
custom-event pricing and is reachable via enablePpeCharging: true or
ACTOR_PPE_ENABLED=1.
[1.0.0] - 2026-08-18
Initial release.
Added
Category crawling of apps.shopify.com with automatic discovery of every
top-level category from /categories when no categories input is supplied.
Non-leaf pages are walked into subcategories up to maxSubcategoryDepth.
One dataset item per app-in-category: name, handle, canonical URL, rating,
review count, pricing label, one-line description, global position, position on
page, page number, category path/name/breadcrumb, and the category's total app
count.
Four-strategy parser ladder (json-ld → embedded-json → data-attributes
→ heuristic) so extraction survives Shopify DOM changes. The winning strategy
is logged and stored on every item as parseStrategy.
DEBUG_<category>_<page>.html dumps to the default key-value store whenever
a listing page yields zero app cards, so selectors can be repaired quickly.
Per-category opportunity metrics — reviews_per_app, top24_avg,
median_sampled, share_under100, top3_share, free_only_share and keyword
cluster counts — written to OUTPUT in the default key-value store and to a
named dataset (category-summaries by default). This is a verified port of the
original Python reference implementation: all 93 categories in the reference
corpus reproduce field-for-field, including Python's banker's rounding.
Cross-category keyword cluster roll-up in OUTPUT, with 13 built-in
regulatory/compliance clusters (GDPR, EU withdrawal, accessibility, e-invoicing,
GPSR, EPR, tax/VAT, fraud, …) and support for user-supplied
keywordClusters: [{ name, regex }].
Optional app detail enrichment (includeAppDetails) for developer, launch
date, pricing plans and category tags, with JSON-LD and DOM fallbacks.
Pay-per-event billing via analyzer-run-start and app-record events.
Prices live in the Apify Console and .actor/pay_per_event.json; no dollar
amount is present in the Actor code.
Apify Proxy support, session rotation, per-session cookies, configurable
retries, and polite concurrency capped at 10 (default 5).
Migration-safe state via Actor.useState, so a restarted run resumes with
its category accumulators intact.
scripts/analyze-local.ts — recompute every metric offline from an exported
dataset JSON and emit leaf_metrics.json, leaf_metrics.csv and
keyword_clusters.json.
Test suite (45 vitest cases): pure parser tests against hand-written
fixtures for each DOM shape, metric parity tests against golden data from the
Python reference, and an HTTP integration test that runs Crawlee against a local
fixture server.
Dockerfile.playwright for the optional useBrowser mode.
Known limitations
Not yet smoke-tested against the live site; see the Limitations and known
risks section of the README.
useBrowser: true requires building from Dockerfile.playwright with
playwright added to dependencies.
Only the default apps.shopify.com locale is crawled.