# Universal Store Catalog Scraper (Shopify/Woo/Generic) (`maninder/store-catalog-scraper`) Actor Extract clean, normalized product-catalog data from any Shopify, WooCommerce, or generic e-commerce storefront. AI-agent / MCP ready: every store comes back in one identical schema. - **URL**: https://apify.com/maninder/store-catalog-scraper.md - **Developed by:** [Maninder Pal Singh](https://apify.com/maninder) (community) - **Categories:** E-commerce - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $4.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Universal Store Catalog Scraper 🛍️ → 🤖 Turn **any Shopify, WooCommerce, or generic e-commerce storefront** into a **clean, normalized product feed** — the same schema for every store, every time. Built **MCP-first** so an AI agent can consume it as a structured catalog tool. This is a **storefront-platform scraper, not a marketplace scraper**. It targets independent merchant stores (each its own site) using **public, non-personal product data only**. The core value is **cross-platform schema consistency**: a Shopify store, a WooCommerce store, and a plain JSON-LD store all come back in *one identical shape*. --- ### ✨ Why this Actor - **One schema, every platform.** Agents don't have to special-case Shopify vs. Woo vs. generic — the output is always the same record. - **Tiered, cheap-first extraction.** Uses fast public JSON endpoints where they exist and only spins up a browser as a last resort, so runs stay fast and cheap. - **Reliable & graceful.** A malformed or blocked store degrades to fewer fields or a skip record — it never fails the whole run. - **Polite by default.** Respects `robots.txt`, rate-limits per domain, honest User-Agent, public data only. --- ### 🧱 How it extracts (tiered engine) | Platform | Primary path | Fallback | |---|---|---| | **Shopify** | Public `/products.json` (paginated, `?limit=250&page=N`) | Generic JSON-LD | | **WooCommerce** | Store API `/wp-json/wc/store/v1/products` | Generic JSON-LD / HTML | | **Generic** | schema.org `Product` JSON-LD | Open Graph → microdata → **Playwright render** | Detection is a single lightweight request (asset/header/global fingerprints, with a cheap endpoint probe when ambiguous) and is overridable via `forcePlatform`. --- ### 📥 Input Two modes — use either or combine them. **Mode 1 — Direct URLs (default, safest)** ```json { "storeUrls": ["https://www.allbirds.com", "https://some-woostore.com/shop"], "maxProductsPerStore": 200, "includeVariants": true, "includeImages": true } ```` **Mode 2 — Discovery (opt-in, capped)** ```json { "discovery": { "enabled": true, "keywords": ["handmade ceramic mugs", "merino wool socks"], "platformHint": "any", "maxStores": 10 }, "maxProductsPerStore": 100 } ``` | Field | Type | Default | Notes | |---|---|---|---| | `storeUrls` | string\[] | `[]` | Store home / collection / product URLs | | `discovery.enabled` | bool | `false` | Opt-in store discovery | | `discovery.keywords` | string\[] | `[]` | Niche/product terms | | `discovery.platformHint` | enum | `any` | `shopify` | `woocommerce` | `any` | | `discovery.maxStores` | int | `10` | **Hard max 50** | | `maxProductsPerStore` | int | `200` | **Hard max 5000** — cost ceiling | | `includeVariants` | bool | `true` | Per-variant array | | `includeImages` | bool | `true` | Image URLs (primary first) | | `includeDescriptionHtml` | bool | `false` | Raw HTML descriptions (bulky) | | `forcePlatform` | enum | `auto` | Override detection | | `respectRobotsTxt` | bool | `true` | Honor robots.txt | | `maxConcurrency` | int | `5` | Stores in parallel | | `proxyConfiguration` | object | Apify auto | — | *** ### 📤 Output — one normalized record per product ```json { "storeUrl": "https://www.allbirds.com", "storeName": "Allbirds", "platform": "shopify", "productId": "6592289505353", "productUrl": "https://www.allbirds.com/products/mens-wool-runners", "title": "Men's Wool Runners", "descriptionText": "Our iconic everyday sneaker made from merino wool...", "brand": "Allbirds", "productType": "Shoes", "categories": ["Shoes"], "price": 110.0, "compareAtPrice": null, "currency": "USD", "onSale": false, "availability": "in_stock", "sku": "WR-NAT-GRY-8", "variants": [ { "variantId": "39472394814025", "title": "Natural Grey / 8", "price": 110.0, "compareAtPrice": null, "sku": "WR-NAT-GRY-8", "availability": "in_stock", "options": { "Color": "Natural Grey", "Size": "8" } } ], "images": ["https://cdn.shopify.com/.../wool-runner-1.jpg"], "tags": ["wool", "runners"], "ratingValue": null, "ratingCount": null, "scrapedAt": "2026-06-30T10:15:42.118Z", "extractionMethod": "shopify:products.json" } ``` **Field guarantees that matter for agents:** - `price` / `compareAtPrice` are always **numbers** (never `"$19.99"`); `currency` is a separate **ISO-4217** code. - `availability` is one of `in_stock` | `out_of_stock` | `preorder` | `unknown`. - `platform` is one of `shopify` | `woocommerce` | `generic`. - `extractionMethod` tells you exactly which path produced the record. - Stores that yield nothing produce a `{ storeUrl, skipped, error }` row instead of disappearing. See [`samples/`](./samples) for a full input + output example per platform. *** ### 🤖 MCP / AI-agent usage Apify exposes every Actor as an **MCP tool**, so an agent can call this scraper directly. Because the input schema is crisp and the output is deterministic and identical across platforms, the agent experience is clean. **Connect** (Apify MCP server): `https://mcp.apify.com` — add `YOUR_USERNAME/store-catalog-scraper` as an available tool. **An agent calls it like this** (conceptual tool call): ```json { "tool": "YOUR_USERNAME/store-catalog-scraper", "input": { "storeUrls": ["https://www.allbirds.com"], "maxProductsPerStore": 50, "includeVariants": true } } ``` The agent then reads the run's **dataset** — every item is one normalized product it can reason over (compare prices across stores, check stock, build a catalog) without per-store parsing logic. > Tip for agent builders: keep `maxProductsPerStore` modest for interactive use, and rely on `platform` + `extractionMethod` to gauge data confidence. *** ### 💸 Pricing (pay-per-event) | Event | Rate (USD) | When | |---|---|---| | **Result (product scraped)** | **$0.004** ($4 / 1,000) | Per normalized product written to the dataset — the main charge | | **Actor start** | $0.02 | Once per run | Platform compute + proxy are **included** (absorbed by the Actor), so you pay only the events above — pricing is predictable regardless of a store's size or speed. **Worked cost example** — scrape **10 stores × 200 products = 2,000 products**: ``` Actor start: 1 × $0.02 = $0.02 Products scraped: 2000 × $0.004 = $8.00 ------------------------------------------- Total ≈ $8.02 (~$0.004 per product) ``` You are only charged for **real products**: stores that fail or return nothing are written to a separate, **non-billed** dataset (`skipped-stores`). *** ### 🔒 Responsible use - **Public, non-personal product data only.** This Actor does **not** log into stores, collect merchant/customer personal data, or access gated content. If a store requires authentication to view products, it is skipped. - **You own compliance.** You are responsible for complying with each target store's Terms of Service and applicable law in your jurisdiction. - **Discovery is best-effort and user-directed.** Mode 2 finds *candidate* public storefronts from your keywords and verifies them before crawling; you remain responsible for what you choose to crawl. It is hard-capped at 50 stores. - **Politeness is built in:** `robots.txt` is respected by default, requests are rate-limited per domain with jitter, and an honest User-Agent is sent. *** ### ⚠️ Limitations - WooCommerce per-variant **prices** aren't exposed by the public Store API without extra calls, so Woo variants surface attribute combinations with `price: null` (product-level price is always populated). - Generic JSON-LD stores vary widely; fields present depend on what the store emits. Missing fields are `null`/`[]`, never invented. - Shopify currency is read from the storefront; if a store hides it, `currency` may be `null`. - The Playwright fallback only triggers when cheap paths return nothing **and** the page looks JS-rendered; escalations are capped per run for cost. *** ### 🧩 Extending Platform modules are cleanly separated (`src/shopify.js`, `src/woocommerce.js`, `src/generic.js`) and all funnel through `src/normalize.js`. Adding **BigCommerce** or **Squarespace** is a drop-in: add a detector fingerprint + an extractor that emits into the shared normalizer. # Actor input Schema ## `storeUrls` (type: `array`): Mode 1 (default, safest). Store homepages, collection URLs, or product URLs. Each entry is treated as its own storefront. ## `discovery` (type: `object`): Mode 2. Best-effort discovery of stores from keywords. Disabled by default. You are responsible for what you crawl. ## `maxProductsPerStore` (type: `integer`): Hard cost ceiling per store. Pagination stops here even if more products exist. ## `includeVariants` (type: `boolean`): Emit the per-variant array (price, sku, options, availability). ## `includeImages` (type: `boolean`): Emit the images array (primary image first). ## `includeDescriptionHtml` (type: `boolean`): Raw HTML descriptions are bulky. Off by default — clean text is always provided in descriptionText. ## `forcePlatform` (type: `string`): Override auto-detection. ## `respectRobotsTxt` (type: `boolean`): Honor each store's robots.txt before requesting a URL. Strongly recommended. ## `maxConcurrency` (type: `integer`): How many stores to process in parallel. Kept low to stay polite per merchant. ## `proxyConfiguration` (type: `object`): Proxy used for all requests. Apify auto proxy by default. ## Actor input object example ```json { "storeUrls": [ "https://www.allbirds.com", "https://www.example-woostore.com" ], "discovery": { "enabled": false, "keywords": [], "platformHint": "any", "maxStores": 10 }, "maxProductsPerStore": 200, "includeVariants": true, "includeImages": true, "includeDescriptionHtml": false, "forcePlatform": "auto", "respectRobotsTxt": true, "maxConcurrency": 5, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `products` (type: `string`): Normalized product records stored in the default dataset. This is the Actor's main output. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "storeUrls": [ "https://www.allbirds.com", "https://www.example-woostore.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("maninder/store-catalog-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "storeUrls": [ "https://www.allbirds.com", "https://www.example-woostore.com", ] } # Run the Actor and wait for it to finish run = client.actor("maninder/store-catalog-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "storeUrls": [ "https://www.allbirds.com", "https://www.example-woostore.com" ] }' | apify call maninder/store-catalog-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=maninder/store-catalog-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Universal Store Catalog Scraper (Shopify/Woo/Generic)", "description": "Extract clean, normalized product-catalog data from any Shopify, WooCommerce, or generic e-commerce storefront. AI-agent / MCP ready: every store comes back in one identical schema.", "version": "1.0", "x-build-id": "K6ozLI00m6uoLCy6R" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/maninder~store-catalog-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-maninder-store-catalog-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/maninder~store-catalog-scraper/runs": { "post": { "operationId": "runs-sync-maninder-store-catalog-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/maninder~store-catalog-scraper/run-sync": { "post": { "operationId": "run-sync-maninder-store-catalog-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "storeUrls": { "title": "Store URLs", "type": "array", "description": "Mode 1 (default, safest). Store homepages, collection URLs, or product URLs. Each entry is treated as its own storefront.", "default": [], "items": { "type": "string" } }, "discovery": { "title": "Discovery (opt-in, capped)", "type": "object", "description": "Mode 2. Best-effort discovery of stores from keywords. Disabled by default. You are responsible for what you crawl.", "default": { "enabled": false, "keywords": [], "platformHint": "any", "maxStores": 10 } }, "maxProductsPerStore": { "title": "Max products per store", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Hard cost ceiling per store. Pagination stops here even if more products exist.", "default": 200 }, "includeVariants": { "title": "Include variants", "type": "boolean", "description": "Emit the per-variant array (price, sku, options, availability).", "default": true }, "includeImages": { "title": "Include images", "type": "boolean", "description": "Emit the images array (primary image first).", "default": true }, "includeDescriptionHtml": { "title": "Include raw HTML descriptions", "type": "boolean", "description": "Raw HTML descriptions are bulky. Off by default — clean text is always provided in descriptionText.", "default": false }, "forcePlatform": { "title": "Force platform", "enum": [ "auto", "shopify", "woocommerce", "generic" ], "type": "string", "description": "Override auto-detection.", "default": "auto" }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Honor each store's robots.txt before requesting a URL. Strongly recommended.", "default": true }, "maxConcurrency": { "title": "Max concurrent stores", "minimum": 1, "maximum": 20, "type": "integer", "description": "How many stores to process in parallel. Kept low to stay polite per merchant.", "default": 5 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy used for all requests. Apify auto proxy by default.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```