# Shopify Product Feed & Price Scraper (`tpa_d/shopify-beauty-radar`) Actor Scrape public Shopify products.json feeds for product catalogs, variants, prices, compare-at discounts, availability, tags, images, and new product snapshots. - **URL**: https://apify.com/tpa\_d/shopify-beauty-radar.md - **Developed by:** [The Product Analyst Data](https://apify.com/tpa_d) (community) - **Categories:** E-commerce - **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 product / variant rows This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Shopify Product Feed & Price Scraper 🛍️📈 Scrape public Shopify product feeds from compatible stores and export clean product, variant, price, availability, tag, image, and discount rows. This Actor reads Shopify’s public `/products.json` catalog endpoint — **no browser, login, proxy, or Shopify admin access required** — and turns it into structured data you can export to JSON, CSV, Excel, Google Sheets, or consume through the Apify API. It works especially well for **competitor product catalog snapshots**, **Shopify price monitoring**, **discount tracking**, **new product drop discovery**, and **variant-level ecommerce research**. Beauty, skincare, and hair-care stores are the default examples, but the Actor can scan any Shopify store that exposes the public product feed. ![Shopify Product Feed & Price Scraper sample results](assets/product-radar-results-preview.svg) ### What you get - **Shopify product feed scraping** — product title, vendor, product type, tags, product URL, image URL, and publish date. - **Variant-level rows** — capture size, shade, bundle, SKU-like variant title, price, compare-at price, and availability at the variant level. - **Price and discount intelligence** — detect real Shopify `compare_at_price` markdowns and compute discount percentage. - **New product and catalog snapshots** — sort by `newest` to identify recently published products. - **Keyword filtering** — scan only matching products such as `serum`, `spf`, `bundle`, `dress`, `supplement`, or any category terms. - **Opportunity scoring** — rank in-stock markdowns and sale opportunities with a bounded `dropOrSaleScore`. - **Structured diagnostics** — invalid, blocked, disabled, or no-match feeds return diagnostic rows instead of disappearing. - **Automation-ready output** — schedule repeated runs and diff datasets for restocks, price changes, new drops, and assortment changes. ### Best use cases - Scrape a Shopify competitor’s public catalog into a spreadsheet. - Monitor competitor prices, compare-at prices, and markdowns. - Track new product drops by sorting products by publish date. - Compare product assortment, tags, product types, vendors, and price bands across stores. - Build daily or weekly Shopify catalog snapshots for alerts and BI workflows. - Research beauty, skincare, hair-care, fashion, supplement, home goods, or other Shopify-powered ecommerce niches. ### Buyer-ready offer > Send me 3-5 Shopify competitor stores. I’ll return a clean product/variant/price/discount spreadsheet with top markdowns and new-product signals. First sample is free. Best buyers: Shopify agencies, DTC brands, ecommerce operators, retention/CRO teams, and analysts who need a repeatable competitor catalog feed. ### Input example ```json { "storeUrls": [ "https://mielleorganics.com", "https://baskandlatherco.com", "https://kaleidoscopehairproducts.com", "https://patternbeauty.com", "https://themanemagic.com" ], "keywords": ["shampoo", "conditioner", "oil", "serum", "leave-in", "curl", "edge", "mask"], "maxProductsPerStore": 100, "minDiscountPercent": 0, "includeVariants": true, "sortBy": "opportunity", "includeDiagnostics": true } ```` | Field | What it controls | |---|---| | `storeUrls` **required** | Shopify store URLs to scan. Bare domains are accepted. Paths are normalized to the store origin. | | `keywords` | Filter products by title, vendor, product type, and tags. Empty = keep all products. | | `maxProductsPerStore` | Product cap per store before variant expansion. | | `minDiscountPercent` | Minimum markdown percent to return. Use `0` for all matches. | | `includeVariants` | `true` = one row per variant; `false` = one row per product using its best-deal variant. | | `sortBy` | `opportunity`, `discount`, `price`, or `newest`. | | `includeDiagnostics` | Include invalid, blocked, failed, or no-match rows for auditability. | ### Sample output Real top row from the sample input: ```json { "rowType": "product", "storeUrl": "https://mielleorganics.com", "productId": 6752188989553, "title": "Mango & Tulsi Nourishing Conditioner", "vendor": "MIELLE", "productType": "Conditioners", "productUrl": "https://mielleorganics.com/products/mango-tulsi-nourishing-conditioner", "imageUrl": "https://cdn.shopify.com/s/files/1/0763/8199/files/250513-Mielle_SHA-Mielle-MT-Conditioner-3.jpg?v=1747750810", "publishedAt": "2022-09-02T17:31:07-05:00", "variantId": 39844163518577, "variantTitle": "Default Title", "price": 5, "compareAtPrice": 14.99, "discountPercent": 67, "available": true, "matchedKeywords": ["conditioner"], "dropOrSaleScore": 72, "scrapedAt": "2026-07-04T02:50:20.087Z" } ``` Example run result from the sample input: | Store | Result | |---|---:| | `mielleorganics.com` | 53 rows | | `baskandlatherco.com` | 38 rows | | `kaleidoscopehairproducts.com` | 41 rows | | `patternbeauty.com` | 59 rows | | `themanemagic.com` | 1 diagnostic row | | **Total** | **192 dataset rows** | The top finding was a **67% off** in-stock Mielle conditioner with a `dropOrSaleScore` of `72`. ### Output fields | Field | Meaning | |---|---| | `rowType` | `product` or `diagnostic`. | | `storeUrl` | Normalized Shopify store origin. | | `productId`, `title`, `vendor`, `productType`, `tags` | Shopify product metadata. | | `productUrl`, `imageUrl` | Product page and primary image. | | `publishedAt` | Shopify publish timestamp. | | `variantId`, `variantTitle` | Specific variant scanned. | | `price`, `compareAtPrice` | Current and compare-at prices parsed to numbers. | | `discountPercent` | Rounded markdown percent when `compareAtPrice > price`. | | `available` | Shopify variant availability boolean. | | `matchedKeywords` | Which configured keywords matched. | | `dropOrSaleScore` | 0–100 opportunity score; deeper, in-stock markdowns rank higher. | | `scrapedAt` | Run timestamp. | ### Diagnostic rows When `includeDiagnostics` is enabled, stores that fail or return no usable feed produce rows like: ```json { "rowType": "diagnostic", "storeUrl": "https://themanemagic.com", "statusCode": 200, "errorType": "parse_error", "message": "Could not parse products.json: Unexpected token '<' ... is not valid JSON", "scrapedAt": "2026-07-04T02:50:20.087Z" } ``` Possible `errorType` values: `invalid_url`, `http_error`, `network_error`, `parse_error`, `invalid_response`, `no_matching_rows`. ### How it works This Actor queries each store’s public Shopify product feed at: ```text /products.json ``` It paginates the feed, normalizes products and variants, applies optional keyword and discount filters, sorts rows, and writes product or diagnostic rows to the default dataset. It does **not** log in, scrape private data, bypass paywalls, or access hidden Shopify admin data. ### Pricing This Actor is configured for **Pay per event** pricing: - `apify-actor-start`: **$0.00005 per run start**. - `product-row`: **$0.001 per real product / variant row**. - Diagnostic rows are emitted for transparency but are **not** charged as value rows. Displayed price: **from $1.00 / 1,000 product or variant rows**. Example: a 1,263-row competitor scan costs about **$1.263** in Actor event charges, plus/minus Apify platform usage depending on the user’s plan/settings. ### Limitations - Works only with stores exposing Shopify’s public `/products.json` endpoint. - Some stores disable, block, or alter the public product feed. - Detects markdowns from `compare_at_price`; coupon/cart-only discounts are not visible. - Inventory is a boolean (`available`), not exact stock count. - Region-specific pricing and hidden/draft products are not included. - A run is a snapshot; schedule repeated runs and compare datasets for true change tracking. - Not affiliated with Shopify or any listed brand. Use responsibly and respect each site’s terms. ### Ecommerce Data Toolkit This Actor is part of a focused ecommerce / retail-intelligence toolkit: | Need | Actor | |---|---| | Shopify product feeds, variants, prices, discounts | [Shopify Product Feed & Price Scraper](https://apify.com/fierce_quoll/shopify-beauty-radar) | | Brand retail footprint and where-to-buy / store-locator rows | [Brand Store Locator & Stockist Scraper](https://apify.com/fierce_quoll/beauty-stockist-radar) | | Salon, stylist, and certified-pro locator rows | [Beauty Salon & Pro Locator Scraper](https://apify.com/fierce_quoll/beauty-pro-locator-radar) | Use the three together to map **what brands sell**, **where products are stocked**, and **which salons/pros carry or certify around professional lines**. ### FAQ #### Does this work on every Shopify store? No. It works when the store exposes the public `/products.json` endpoint. Some stores disable or alter that endpoint. #### Can I monitor changes over time? Yes. Schedule this Actor daily or hourly, export datasets, and compare new runs against previous runs. #### Does it require proxies? No proxy is required for the default public-feed approach. #### Can I export to CSV or Excel? Yes. Apify datasets can be exported to JSON, CSV, Excel, XML, RSS, or consumed through API integrations. ### Quality / verification This project now ships with a reusable sample input and standard local checks: ```bash npm run check cp examples/sample-input.json storage/key_value_stores/default/INPUT.json CRAWLEE_STORAGE_DIR=./storage APIFY_LOCAL_STORAGE_DIR=./storage node src/main.js ``` The buyer demo package lives at: ```text /opt/data/apify-projects/outreach/shopify-product-feed-demo.zip ``` ### API usage Run from the Apify CLI: ```bash apify call fierce_quoll/shopify-beauty-radar --input-file examples/live-input.json ``` Or use the API tab on the Actor page for language-specific snippets. ### Local development ```bash npm install npm test /opt/data/bin/apify validate-schema .actor/input_schema.json mkdir -p storage/key_value_stores/default cp examples/live-input.json storage/key_value_stores/default/INPUT.json CRAWLEE_STORAGE_DIR=./storage APIFY_LOCAL_STORAGE_DIR=./storage node src/main.js ``` # Actor input Schema ## `storeUrls` (type: `array`): Public Shopify store URLs to scan. Bare domains like 'examplestore.com' are accepted and normalized. The Actor reads each store's public /products.json feed when available. ## `keywords` (type: `array`): Case-insensitive keywords matched against product title, vendor, product type, and tags. Leave empty to include every product. Matched keywords are reported per row. ## `maxProductsPerStore` (type: `integer`): Maximum number of products to fetch from each store before variant expansion. Pagination stops once this many products are collected. ## `minDiscountPercent` (type: `number`): Only emit rows whose discount (compare-at vs current price) is at least this percent. Use 0 to include full-price products too. ## `includeVariants` (type: `boolean`): When enabled, emit one row per product variant. When disabled, emit one row per product using its best-deal (highest-discount) variant. ## `sortBy` (type: `string`): opportunity = drop/sale score; discount = discount percent; price = current price ascending; newest = most recently published first. Sorting is applied within each store. ## `includeDiagnostics` (type: `boolean`): Emit diagnostic rows (rowType='diagnostic') when a store is unreachable, returns a non-200 status, blocks the /products.json feed, or yields no matching products. ## Actor input object example ```json { "storeUrls": [ "https://mielleorganics.com", "https://baskandlatherco.com", "https://kaleidoscopehairproducts.com", "https://patternbeauty.com" ], "keywords": [ "serum", "cleanser", "spf", "lip", "bundle" ], "maxProductsPerStore": 250, "minDiscountPercent": 0, "includeVariants": true, "sortBy": "opportunity", "includeDiagnostics": true } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "storeUrls": [ "https://mielleorganics.com", "https://baskandlatherco.com", "https://kaleidoscopehairproducts.com", "https://patternbeauty.com" ], "keywords": [ "serum", "cleanser", "spf", "lip", "bundle" ] }; // Run the Actor and wait for it to finish const run = await client.actor("tpa_d/shopify-beauty-radar").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "storeUrls": [ "https://mielleorganics.com", "https://baskandlatherco.com", "https://kaleidoscopehairproducts.com", "https://patternbeauty.com", ], "keywords": [ "serum", "cleanser", "spf", "lip", "bundle", ], } # Run the Actor and wait for it to finish run = client.actor("tpa_d/shopify-beauty-radar").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "storeUrls": [ "https://mielleorganics.com", "https://baskandlatherco.com", "https://kaleidoscopehairproducts.com", "https://patternbeauty.com" ], "keywords": [ "serum", "cleanser", "spf", "lip", "bundle" ] }' | apify call tpa_d/shopify-beauty-radar --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=tpa_d/shopify-beauty-radar", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Shopify Product Feed & Price Scraper", "description": "Scrape public Shopify products.json feeds for product catalogs, variants, prices, compare-at discounts, availability, tags, images, and new product snapshots.", "version": "0.1", "x-build-id": "k0LVOQrtYx8FFyO7q" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/tpa_d~shopify-beauty-radar/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-tpa_d-shopify-beauty-radar", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/tpa_d~shopify-beauty-radar/runs": { "post": { "operationId": "runs-sync-tpa_d-shopify-beauty-radar", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/tpa_d~shopify-beauty-radar/run-sync": { "post": { "operationId": "run-sync-tpa_d-shopify-beauty-radar", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "storeUrls" ], "properties": { "storeUrls": { "title": "Shopify store URLs", "type": "array", "description": "Public Shopify store URLs to scan. Bare domains like 'examplestore.com' are accepted and normalized. The Actor reads each store's public /products.json feed when available.", "items": { "type": "string" } }, "keywords": { "title": "Keywords (optional)", "type": "array", "description": "Case-insensitive keywords matched against product title, vendor, product type, and tags. Leave empty to include every product. Matched keywords are reported per row.", "items": { "type": "string" } }, "maxProductsPerStore": { "title": "Max products per store", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of products to fetch from each store before variant expansion. Pagination stops once this many products are collected.", "default": 250 }, "minDiscountPercent": { "title": "Minimum discount percent", "minimum": 0, "maximum": 100, "type": "number", "description": "Only emit rows whose discount (compare-at vs current price) is at least this percent. Use 0 to include full-price products too.", "default": 0 }, "includeVariants": { "title": "Include all variants", "type": "boolean", "description": "When enabled, emit one row per product variant. When disabled, emit one row per product using its best-deal (highest-discount) variant.", "default": true }, "sortBy": { "title": "Sort products by", "enum": [ "opportunity", "discount", "price", "newest" ], "type": "string", "description": "opportunity = drop/sale score; discount = discount percent; price = current price ascending; newest = most recently published first. Sorting is applied within each store.", "default": "opportunity" }, "includeDiagnostics": { "title": "Include diagnostics", "type": "boolean", "description": "Emit diagnostic rows (rowType='diagnostic') when a store is unreachable, returns a non-200 status, blocks the /products.json feed, or yields no matching products.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```