# Shopify Store Scraper (`scrapepilotapi/shopify-store-scraper`) Actor - **URL**: https://apify.com/scrapepilotapi/shopify-store-scraper.md - **Developed by:** [ScrapePilot](https://apify.com/scrapepilotapi) (community) - **Categories:** Automation, E-commerce, Lead generation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $3.99 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Shopify Store Scraper A powerful Apify Actor that extracts comprehensive data from Shopify stores, including products, collections, themes, installed apps, and optional product reviews from Yotpo. Perfect for competitive analysis, market research, product monitoring, and e-commerce intelligence gathering. ### Why Choose Us? - **🔄 Smart Proxy Fallback**: Automatically switches between no proxy, datacenter, and residential proxies with intelligent retry logic to ensure successful scraping - **📊 Comprehensive Data Extraction**: Captures everything from store metadata to product details, pricing, inventory, and customer reviews - **⚡ High Performance**: Async/await architecture ensures fast concurrent data fetching - **🛡️ Robust Error Handling**: Gracefully handles failures and continues processing remaining stores - **💾 Live Data Saving**: Results are saved in real-time to Apify dataset, so you don't lose data if the actor stops - **📝 Detailed Logging**: Comprehensive logging keeps you informed throughout the scraping process ### Key Features - **Store Metadata**: Extracts store domain, shop name, country, currency, locale, and theme information - **Product Data**: Fetches all products with variants, prices, images, descriptions, and availability - **Collections**: Retrieves all product collections and their organization - **Theme Information**: Identifies theme name, ID, and version - **Installed Apps**: Detects Shopify apps installed on the store (e.g., Elevar, JsEncrypt) - **Pricing Analytics**: Calculates min, max, and average prices across all products - **Inventory Stats**: Provides inventory availability percentages and variant counts - **Product Reviews**: Optional extraction of Yotpo reviews for top products (author, rating, content, dates) - **Top Products**: Identifies newest products with optional review data - **Smart Proxy Management**: Intelligent fallback system ensures requests succeed even when blocked ### Input The actor accepts the following input parameters: #### Input Schema ```json { "startUrls": [ "https://www.allbirds.com", "https://example-store.myshopify.com" ], "extractReviews": true, "proxyConfiguration": { "useApifyProxy": false } } ```` #### Input Fields | Field | Type | Required | Description | |-------|------|----------|-------------| | **startUrls** | `array` | ✅ Yes | List of Shopify store URLs to scrape. Supports bulk input with multiple store URLs. Example: `["https://www.allbirds.com"]` | | **extractReviews** | `boolean` | ❌ No | Enable to extract detailed product reviews from Yotpo for top 10 newest products. Default: `true`. Note: Enabling this will slow down the scraping process. | | **proxyConfiguration** | `object` | ❌ No | Apify proxy configuration. If not provided, the actor will attempt direct requests first. If blocked, it automatically falls back to datacenter proxy, then residential proxy with 3 retries. | #### Input Examples **Basic Usage (No Proxy)** ```json { "startUrls": ["https://www.allbirds.com"], "extractReviews": true } ``` **With Proxy Enabled** ```json { "startUrls": ["https://www.allbirds.com"], "extractReviews": true, "proxyConfiguration": { "useApifyProxy": true } } ``` **Multiple Stores** ```json { "startUrls": [ "https://www.allbirds.com", "https://example-store.myshopify.com", "https://another-store.com" ], "extractReviews": false } ``` ### Output The actor outputs structured JSON data to the Apify dataset. Each store is saved as a separate dataset item. #### Output Schema ```json { "store_domain": "www.allbirds.com", "store_url": "https://www.allbirds.com", "shop_name": "shop11044168.myshopify.com", "country": "US", "currency": "USD", "locale": "en-US", "theme_name": "rc-cm-2025-12-26_20-35 Update", "theme_id": 128596836432, "theme_version": "1.131.2", "apps_installed": ["Elevar Conversion Tracking", "JsEncrypt"], "apps_by_category": { "Analytics": ["Elevar Conversion Tracking"], "Other": ["JsEncrypt"] }, "apps_count": 2, "total_products": 681, "total_variants": 7276, "collections": ["Accessories", "Shoes", "Apparel", ...], "collections_count": 250, "categories": ["Accessories", "Apparel", "Shoes", "Socks"], "vendors": ["Allbirds"], "price_min": 2.0, "price_avg": 79.34, "price_max": 500.0, "products_on_sale": 681, "total_variants_available": 5647, "inventory_availability_pct": 77.6, "estimated_launch_date": "2018-11-12", "oldest_product_date": "2018-11-12", "newest_product_date": "2025-11-06", "products_with_images": 680, "avg_images_per_product": 4.5, "products_with_description": 681, "top_products": [ { "title": "Allbirds Laces - Warm Red - Round", "url": "https://www.allbirds.com/products/allbirds-laces-warm-red-round", "price": 8.0, "reviews": 0, "rating": 0.0 } ], "detailed_reviews": [ { "product_url": "https://www.allbirds.com/products/...", "product_id": "7242343809104", "reviews_count": 0, "avg_rating": 0.0, "reviews": [] } ], "scraped_at": "2025-12-29T07:34:23.715157Z", "scrape_duration_sec": 15.71, "api_blocked": false } ``` #### Output Fields | Field | Type | Description | |-------|------|-------------| | **store\_domain** | `string` | Store domain name (without protocol) | | **store\_url** | `string` | Full store URL | | **shop\_name** | `string` | Shopify shop name (e.g., `shop12345.myshopify.com`) | | **country** | `string` | Store country code | | **currency** | `string` | Store currency code | | **locale** | `string` | Store locale (e.g., `en-US`) | | **theme\_name** | `string` | Active theme name | | **theme\_id** | `integer` | Shopify theme ID | | **theme\_version** | `string` | Theme version number | | **apps\_installed** | `array` | List of detected Shopify apps | | **apps\_by\_category** | `object` | Apps grouped by category | | **apps\_count** | `integer` | Total number of installed apps | | **total\_products** | `integer` | Total number of products in store | | **total\_variants** | `integer` | Total number of product variants | | **collections** | `array` | List of all collection titles | | **collections\_count** | `integer` | Total number of collections | | **categories** | `array` | List of product categories | | **vendors** | `array` | List of product vendors | | **price\_min** | `float` | Minimum product price | | **price\_avg** | `float` | Average product price | | **price\_max** | `float` | Maximum product price | | **total\_variants\_available** | `integer` | Number of available variants | | **inventory\_availability\_pct** | `float` | Percentage of variants in stock | | **estimated\_launch\_date** | `string` | Date of oldest product (store launch estimate) | | **oldest\_product\_date** | `string` | Creation date of oldest product | | **newest\_product\_date** | `string` | Creation date of newest product | | **products\_with\_images** | `integer` | Number of products with images | | **avg\_images\_per\_product** | `float` | Average number of images per product | | **products\_with\_description** | `integer` | Number of products with descriptions | | **top\_products** | `array` | Top 10 newest products with basic info and optional reviews | | **detailed\_reviews** | `array` | Detailed review data (only if `extractReviews` is enabled) | | **scraped\_at** | `string` | ISO timestamp of when scraping completed | | **scrape\_duration\_sec** | `float` | Time taken to scrape the store in seconds | | **api\_blocked** | `boolean` | Whether the Shopify API was blocked | ### 🚀 How to Use the Actor (via Apify Console) 1. **Log in** to [Apify Console](https://console.apify.com) and navigate to **Actors** 2. **Find** the `shopify-store-scraper` actor and click on it 3. **Configure Inputs**: - Add one or more Shopify store URLs in the `startUrls` field - Toggle `extractReviews` to enable/disable review extraction (default: enabled) - Optionally configure proxy settings in `proxyConfiguration` 4. **Run the Actor**: Click the **Start** button to begin scraping 5. **Monitor Progress**: Watch real-time logs to track scraping progress and proxy fallback events 6. **Access Results**: Once complete, go to the **OUTPUT** tab to view scraped data 7. **Export Data**: Download results as JSON or CSV, or access via Apify API ### Best Use Cases - **🔍 Competitive Analysis**: Monitor competitor stores, pricing strategies, and product catalogs - **📈 Market Research**: Analyze market trends, product categories, and vendor distribution - **💰 Price Monitoring**: Track pricing changes across multiple Shopify stores - **📦 Inventory Tracking**: Monitor product availability and inventory levels - **🎨 Theme Research**: Identify popular Shopify themes and their usage - **🔌 App Discovery**: Find which Shopify apps are commonly used by stores - **⭐ Review Analysis**: Extract and analyze customer reviews for product insights - **📊 E-commerce Intelligence**: Build comprehensive databases of Shopify store information ### Frequently Asked Questions #### How long does it take to scrape a store? Scraping time depends on the store size: - Small stores (< 100 products): ~5-10 seconds - Medium stores (100-500 products): ~10-30 seconds - Large stores (500+ products): ~30-60+ seconds - With review extraction enabled: Add ~2-5 seconds per product reviewed #### What happens if a store blocks my requests? The actor implements intelligent proxy fallback: 1. First attempts without proxy 2. If blocked, switches to datacenter proxy 3. If still blocked, switches to residential proxy 4. Retries up to 3 times with residential proxy 5. Once residential proxy is used, it sticks with it for all remaining requests #### Can I scrape multiple stores at once? Yes! Simply add multiple URLs to the `startUrls` array. The actor processes them sequentially to avoid overwhelming servers. #### Do I need to enable proxy? No, proxy is optional. The actor works without proxy, but enabling it can help avoid rate limits and blocks, especially for large-scale scraping. #### What if review extraction fails for a product? The actor continues processing other products even if review extraction fails for some. Failed products will have `reviews: 0` and `rating: 0.0` in the output. #### Can I customize which products get reviews extracted? Currently, the actor extracts reviews for the top 10 newest products. This is a fixed limit to balance speed and comprehensiveness. #### What data format is the output? Output is in JSON format, saved to Apify dataset. You can export it as JSON, CSV, or access it via Apify API. #### Does the actor work with all Shopify stores? Yes, the actor works with any publicly accessible Shopify store. It uses Shopify's public JSON APIs (`/products.json`, `/collections.json`, `/meta.json`) which are available on all Shopify stores. ### Support and Feedback 💬 **For custom solutions or feature requests**, contact us at **dev.scraperengine@gmail.com** We're always looking to improve! If you encounter any issues or have suggestions, please reach out. ### Cautions ⚠️ **Important Legal and Ethical Considerations**: - Data is collected only from **publicly available sources** (Shopify's public JSON APIs) - No data is taken from private accounts or password-protected content - The end user is **responsible for ensuring legal compliance** with: - Local data protection laws (GDPR, CCPA, etc.) - Terms of service of the target websites - Copyright and intellectual property laws - Spam and privacy regulations - **Respect rate limits**: The actor includes delays between requests to be respectful - **Use responsibly**: Only scrape data you have permission to access or that is publicly available - **Review terms of service**: Always check the target store's terms of service before scraping *** **Built with ❤️ using Apify Platform** # Actor input Schema ## `startUrls` (type: `array`): List one or more Shopify store URLs (e.g., https://www.allbirds.com). ## `extractReviews` (type: `boolean`): Enable to extract detailed product reviews from Yotpo (slower but more comprehensive). ## `proxyConfiguration` (type: `object`): Choose which proxies to use. If Shopify rejects the proxy, a residential proxy will be used as a fallback. ## Actor input object example ```json { "startUrls": [ "https://www.allbirds.com" ], "extractReviews": true, "proxyConfiguration": { "useApifyProxy": false } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://www.allbirds.com" ], "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("scrapepilotapi/shopify-store-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://www.allbirds.com"], "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("scrapepilotapi/shopify-store-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://www.allbirds.com" ], "proxyConfiguration": { "useApifyProxy": false } }' | apify call scrapepilotapi/shopify-store-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapepilotapi/shopify-store-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Shopify Store Scraper", "version": "0.1", "x-build-id": "hYUeLFLXoGS504hgp" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapepilotapi~shopify-store-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapepilotapi-shopify-store-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapepilotapi~shopify-store-scraper/runs": { "post": { "operationId": "runs-sync-scrapepilotapi-shopify-store-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapepilotapi~shopify-store-scraper/run-sync": { "post": { "operationId": "run-sync-scrapepilotapi-shopify-store-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Shopify Store URLs", "type": "array", "description": "List one or more Shopify store URLs (e.g., https://www.allbirds.com).", "items": { "type": "string" } }, "extractReviews": { "title": "Extract Product Reviews", "type": "boolean", "description": "Enable to extract detailed product reviews from Yotpo (slower but more comprehensive).", "default": true }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Choose which proxies to use. If Shopify rejects the proxy, a residential proxy will be used as a fallback." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```