Shopify Competitor Intelligence

Generate a complete competitor intelligence report for one or more Shopify stores.

This Actor crawls each store's public endpoints and homepage HTML to extract:

• Store info — name, domain, country, currency, language, theme, theme developer
• Products — via /products.json pagination (titles, prices, compare-at, discounts, availability, variants, images)
• Collections — via /collections.json when available
• SEO — title, meta description, Open Graph, Twitter Cards, JSON-LD, canonical, robots.txt, sitemap
• Technology — analytics, pixels, payments, chat widgets, marketing tools, CDN, JS frameworks, cookie banners
• Content pages — blog, FAQ, about, contact, policies (from nav/footer links)

Features

• TypeScript + Crawlee CheerioCrawler (fast HTTP, no browser required)
• Multi-store batch analysis
• Product pagination with configurable caps (maxProducts, maxProductPages)
• Deduplication of product IDs/handles
• Rate limiting via requestDelayMs → maxRequestsPerMinute
• Apify Proxy support
• Structured JSON dataset (one report per store)
• CSV + Markdown summaries in the key-value store

Installation / local run

cd shopify-competitor-intelligence
npm install
npm run build

Create input:

# storage/key_value_stores/default/INPUT.json
{
  "urls": ["https://www.allbirds.com"],
  "maxProducts": 50,
  "proxyConfiguration": { "useApifyProxy": false }
}

npm start
# or
apify run

Input

Field	Type	Default	Description
urls	string[]	required	Shopify store homepage URLs
maxProducts	integer	100	Max products per store (max 2500)
maxProductPages	integer	5	Max products.json pages (250/page)
includeProducts	boolean	true	Fetch products
includeCollections	boolean	true	Fetch collections
includeContentPages	boolean	true	Detect content pages from homepage links
maxConcurrency	integer	3	Max concurrent requests
minConcurrency	integer	1	Min concurrent requests
requestDelayMs	integer	300	Delay used to derive rate limit
maxRequestRetries	integer	3	HTTP retries
proxyConfiguration	object	Apify Proxy on	Proxy settings

Example input

{
    "urls": ["https://www.allbirds.com", "https://www.brooklinen.com"],
    "maxProducts": 100,
    "maxProductPages": 5,
    "maxConcurrency": 3,
    "includeProducts": true,
    "includeCollections": true,
    "includeContentPages": true
}

Output

Dataset

One item per store — full structured CompetitorReport object (store, products, collections, SEO, technology, content, errors).

Key-value store

Key	Description
COMPETITOR_REPORT.csv	Flattened summary table
COMPETITOR_REPORT.md	Human-readable Markdown report
OUTPUT	Run summary JSON

Example output fields

{
    "store": {
        "storeName": "Allbirds",
        "domain": "allbirds.com",
        "currency": "USD",
        "theme": { "name": "...", "developer": "..." }
    },
    "products": {
        "totalProducts": 100,
        "vendors": ["Allbirds"],
        "productTypes": ["Shoes"],
        "priceRange": { "min": 18, "max": 165, "average": 88.5 },
        "items": [/* product details */]
    },
    "collections": { "totalCollections": 20, "items": [] },
    "seo": { "title": "...", "metaDescription": "...", "openGraph": {} },
    "technology": { "analytics": [], "pixels": [], "marketing": [] },
    "content": { "hasBlog": true, "hasFaq": true, "pages": [] },
    "scrapedAt": "2026-07-13T12:00:00.000Z",
    "errors": []
}

How it works

For each store URL the Actor enqueues:

1. Homepage HTML — SEO, theme, tech stack, content links, store name
2. /robots.txt + sitemap probe
3. /cart.js — currency
4. /products.json?page=N&limit=250 — paginated products (optional)
5. /collections.json — collections (optional)

Results are aggregated in memory per store origin, then pushed as a single dataset item.

Limitations

• Works best on public Shopify storefronts that expose products.json / collections.json
• Headless / Hydrogen / heavily bot-protected stores may block or hide JSON endpoints
• Technology detection is signature-based (not a full Wappalyzer crawl of every asset)
• Theme developer is inferred from theme name/schema when possible
• Product catalog is capped by maxProducts and maxProductPages
• Does not log in or access private admin APIs

FAQ

Does this work only on Shopify?
It targets Shopify public endpoints and theme markers. Non-Shopify sites will return partial data (homepage SEO/tech only) or errors for JSON endpoints.

Why are some products missing?
Increase maxProducts / maxProductPages, or the store may not expose products.json.

Do I need a proxy?
Recommended on the Apify platform for reliability. Locally you can set "useApifyProxy": false.

How is pricing charged?
Pay-per-event: each store report (dataset item) is charged as one result.

Development

npm install
npm run lint
npm run format
npm run build
npm test

Changelog

1.0.0

• Initial release
• Multi-store competitor intelligence reports
• Products, collections, SEO, technology, content detection
• CSV + Markdown exports

License

ISC