# Shopify Products Scraper (`competent_clarinet/shopify-products-scraper`) Actor Scrapes products from public Shopify stores via the /products.json endpoint. Supports multiple stores, per-source limits, round-robin concurrency, direct-first HTTP with Apify Proxy fallback, and per-host rate limiting. - **URL**: https://apify.com/competent\_clarinet/shopify-products-scraper.md - **Developed by:** [Man Mohit verma](https://apify.com/competent_clarinet) (community) - **Categories:** E-commerce, Lead generation, Integrations - **Stats:** 3 total users, 2 monthly users, 88.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### What does Shopify Products Scraper do? Shopify Products Scraper collects product data from public Shopify stores in a single run. It reads each store's public `products.json` feed, supports multiple stores at once, and saves structured product records to an Apify dataset. Use it to build catalogs, monitor competitors, enrich analytics pipelines, or export Shopify product data at scale. ### Features - Scrape **multiple Shopify stores** in one Actor run - Set a **product limit per store** - Control **parallelism** across stores with round-robin scheduling - Limit **requests per host** to reduce rate-limit errors - Choose between **short** (condensed) or **full** (complete Shopify JSON) output - Optional **Apify Proxy support** for stores that block direct requests (disabled by default) ### Input Configure the Actor in the **Input** tab. Main fields: | Field | Description | |-------|-------------| | `websites` | **Required.** One or more Shopify store URLs. | | `limitPerSource` | Maximum products to scrape per store (default: `250`). | | `maxConcurrency` | Max parallel page requests across stores (default: `5`). | | `maxRequestsPerSecond` | Per-host request rate limit (default: `2`). Lower this if you see HTTP 429 errors. | | `outputType` | `short` for a compact schema, or `full` for complete product JSON. | | `proxyConfiguration` | Optional. Direct requests are tried first; proxy is used as fallback. | #### Website examples Simple URL list: ```json [ "https://www.allbirds.com", "https://gymshark.com" ] ```` URLs with optional source IDs (included in output as `sourceId`): ```json [ { "id": "allbirds", "url": "https://www.allbirds.com" }, { "id": "gymshark", "url": "gymshark.com" } ] ``` Store URLs are normalized automatically. Duplicate URLs are removed. ### Output Results are saved to the default dataset. You can download them as **JSON, CSV, Excel, HTML, XML, or RSS** from the run's **Storage** tab. The Output tab includes **Short** and **Full** views that match the `outputType` input setting. #### Short output (`outputType: "short"`) Compact records for reporting and catalog overviews. Main fields include: | Field | Description | |-------|-------------| | `store` | Store hostname | | `title` | Product title | | `url` | Product page URL | | `vendor` | Product vendor | | `priceMin` / `priceMax` | Variant price range | | `available` | Whether any variant is available | | `featuredImage` | Primary image URL | | `tags` | Product tags | | `sourceId` | Your optional source identifier | #### Full output (`outputType: "full"`) Complete Shopify product objects with scrape metadata (`store`, `sourceUrl`, `scrapedAt`, `sourceId`) plus standard fields such as `variants`, `images`, and `options`. #### Output example (short) ```json { "store": "www.allbirds.com", "title": "Wool Runner", "url": "https://www.allbirds.com/products/wool-runner", "vendor": "Allbirds", "priceMin": 98, "priceMax": 98, "available": true, "featuredImage": "https://cdn.shopify.com/...", "tags": ["shoes", "wool"], "sourceId": "allbirds" } ``` ### Tips - Start with a low `limitPerSource` when testing new stores. - If a store blocks requests, enable **Apify Proxy** in `proxyConfiguration`. The Actor tries direct access first and switches to proxy only when needed. - Lower `maxRequestsPerSecond` if you encounter rate limiting (HTTP 429). - Use `outputType: "short"` for dashboards and exports; use `full` when you need raw Shopify product structure. ### Limitations - Works with **public** Shopify stores that expose the standard `products.json` endpoint. - Some stores may block scraping, return incomplete data, or require proxy access. - This Actor scrapes product catalog data only; it does not log in to Shopify Admin. # Actor input Schema ## `websites` (type: `array`): List of Shopify store URLs as strings (e.g. "https://store.com") or objects with "id" and "url" keys. When an id is provided, it is included in the output as sourceId. ## `limitPerSource` (type: `integer`): Maximum number of products to scrape from each website. ## `maxConcurrency` (type: `integer`): Maximum number of page requests to run in parallel across websites. Requests are scheduled in round-robin order across sources. ## `requestTimeoutSecs` (type: `integer`): HTTP request timeout in seconds for each products.json request. ## `maxRequestRetries` (type: `integer`): Number of times to retry a failed products.json request before skipping the page. ## `maxRequestsPerSecond` (type: `number`): Maximum number of HTTP requests per second for each Shopify host. Lower this value if you encounter 429 rate limit errors. ## `outputType` (type: `string`): Use "short" for a condensed product schema, or "full" for the complete Shopify product JSON with camelCase keys. The dataset Output tab provides matching Short and Full views. ## `proxyConfiguration` (type: `object`): Optional fallback proxies. Direct requests are tried first; proxy is used only after a direct request fails for a store. ## Actor input object example ```json { "websites": [ "https://www.allbirds.com" ], "limitPerSource": 250, "maxConcurrency": 5, "requestTimeoutSecs": 60, "maxRequestRetries": 3, "maxRequestsPerSecond": 2, "outputType": "full", "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ``` # Actor output Schema ## `productsShort` (type: `string`): Dataset items in the condensed short schema when outputType is short. ## `productsFull` (type: `string`): Dataset items with complete Shopify product JSON when outputType is full. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "websites": [ "https://www.allbirds.com" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("competent_clarinet/shopify-products-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "websites": ["https://www.allbirds.com"], "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], }, } # Run the Actor and wait for it to finish run = client.actor("competent_clarinet/shopify-products-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "websites": [ "https://www.allbirds.com" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }' | apify call competent_clarinet/shopify-products-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=competent_clarinet/shopify-products-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Shopify Products Scraper", "description": "Scrapes products from public Shopify stores via the /products.json endpoint. Supports multiple stores, per-source limits, round-robin concurrency, direct-first HTTP with Apify Proxy fallback, and per-host rate limiting.", "version": "0.1", "x-build-id": "v8GxIFL23BvSNsVJS" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/competent_clarinet~shopify-products-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-competent_clarinet-shopify-products-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/competent_clarinet~shopify-products-scraper/runs": { "post": { "operationId": "runs-sync-competent_clarinet-shopify-products-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/competent_clarinet~shopify-products-scraper/run-sync": { "post": { "operationId": "run-sync-competent_clarinet-shopify-products-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "websites" ], "properties": { "websites": { "title": "Shopify websites", "type": "array", "description": "List of Shopify store URLs as strings (e.g. \"https://store.com\") or objects with \"id\" and \"url\" keys. When an id is provided, it is included in the output as sourceId." }, "limitPerSource": { "title": "Products per website", "minimum": 1, "maximum": 100000, "type": "integer", "description": "Maximum number of products to scrape from each website.", "default": 250 }, "maxConcurrency": { "title": "Max concurrent scrapes", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of page requests to run in parallel across websites. Requests are scheduled in round-robin order across sources.", "default": 5 }, "requestTimeoutSecs": { "title": "Request timeout (seconds)", "minimum": 5, "maximum": 300, "type": "integer", "description": "HTTP request timeout in seconds for each products.json request.", "default": 60 }, "maxRequestRetries": { "title": "Max request retries", "minimum": 0, "maximum": 10, "type": "integer", "description": "Number of times to retry a failed products.json request before skipping the page.", "default": 3 }, "maxRequestsPerSecond": { "title": "Max requests per second (per host)", "minimum": 0.1, "maximum": 100, "type": "number", "description": "Maximum number of HTTP requests per second for each Shopify host. Lower this value if you encounter 429 rate limit errors.", "default": 2 }, "outputType": { "title": "Output type", "enum": [ "short", "full" ], "type": "string", "description": "Use \"short\" for a condensed product schema, or \"full\" for the complete Shopify product JSON with camelCase keys. The dataset Output tab provides matching Short and Full views.", "default": "full" }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Optional fallback proxies. Direct requests are tried first; proxy is used only after a direct request fails for a store." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```