Shopify Universal Scraper

An Apify Actor that scrapes any Shopify store's entire product catalog — including stores with 12,500+ products.

Features

• Full catalog extraction — Automatically paginates through /products.json and reconciles with XML sitemaps for stores exceeding Shopify's 12,500-product pagination ceiling.
• Rich product data — Extracts titles, descriptions (HTML + plain text), images, variants, pricing, stock status, tags, options, weight, and more.
• Anti-bot bypass — Uses curl_cffi with browser impersonation to bypass Shopify's bot detection.
• Proxy support — Works with Apify Proxy (residential, datacenter) or your own custom proxy URL.
• Batch output — Pushes results to an Apify Dataset in batches for efficient streaming.

Input

Field	Type	Required	Default	Description
domain	string	✅	—	Shopify store domain (e.g. gymshark.com)
maxProducts	integer	❌	0	Max products to scrape (0 = all)
proxy	object	❌	Apify Proxy	Apify proxy configuration
customProxyUrl	string	❌	—	Custom proxy URL (overrides Apify Proxy)
concurrency	integer	❌	10	Concurrent requests for sitemap reconciliation
impersonate	string	❌	chrome120	Browser to impersonate via curl_cffi

Example Input

{
    "domain": "gymshark.com",
    "maxProducts": 100,
    "proxy": {
        "useApifyProxy": true
    },
    "concurrency": 10,
    "impersonate": "chrome120"
}

Output

Each product in the dataset has this structure:

{
    "store": "gymshark.com",
    "productId": 123456789,
    "handle": "speed-t-shirt",
    "vendor": "Gymshark",
    "title": "Speed T-Shirt",
    "url": "https://gymshark.com/products/speed-t-shirt",
    "featuredImage": "https://cdn.shopify.com/...",
    "imageUrls": ["..."],
    "imageCount": 5,
    "currency": "USD",
    "priceMin": 30.0,
    "priceMax": 35.0,
    "compareAtPrice": 40.0,
    "onSale": true,
    "available": true,
    "fullyOutOfStock": false,
    "variantCount": 12,
    "variants": [{ "..." }],
    "options": ["Size", "Color"],
    "productOptions": [{ "..." }],
    "productType": "T-Shirts",
    "tags": ["new-arrivals", "men"],
    "description": "Lightweight training tee...",
    "descriptionHtml": "<p>Lightweight training tee...</p>",
    "createdAt": "2024-01-15T10:00:00Z",
    "publishedAt": "2024-01-15T10:00:00Z",
    "updatedAt": "2024-06-01T08:30:00Z"
}

A summary is also saved to the default Key-Value Store under the key SUMMARY:

{
    "store": "gymshark.com",
    "totalProducts": 1842,
    "maxProductsRequested": 0,
    "proxyUsed": true,
    "paginationCapHit": false
}

How It Works

1. Phase 1 — Bulk Pagination: Fetches products from /products.json?limit=250&page=N up to page 50 (Shopify's hard limit of 12,500 products).
2. Phase 2 — Sitemap Reconciliation (only if Phase 1 hits the cap): Parses the store's XML sitemaps to discover all product handles, then fetches any products missed by pagination via individual /products/{handle}.json requests.

Local Development

The original shopify.py script is preserved for local development and testing:

pip install -r requirements.txt
python shopify.py

Deployment

# Install the Apify CLI
npm install -g apify-cli

# Login to your Apify account
apify login

# Deploy the actor
apify push

License

ISC