# E-commerce Store Catalog Scraper (`itsyas/store-catalog-scraper`) Actor 92% success. Give it any store URL — get the full product catalog as clean JSON: products, variants, prices, sale prices, stock and images. Auto-detects Shopify, WooCommerce, PrestaShop and Magento. - **URL**: https://apify.com/itsyas/store-catalog-scraper.md - **Developed by:** [Angel Sanchez](https://apify.com/itsyas) (community) - **Categories:** E-commerce, Automation, Developer tools - **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 product extracteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Any E-commerce Store Catalog Scraper **Give it a store URL. Get the entire product catalog as clean, structured JSON** — products, variants, per-variant prices, sale prices, stock and images. Most scrapers only work on one platform, and only on the big, famous marketplaces. This one **auto-detects the store's platform** (Shopify, WooCommerce, PrestaShop, Magento) and extracts the full catalog of *any* shop — including the small and mid-size stores every other tool ignores. Custom-built stores with no recognisable platform? An **AI agent** explores the site and builds an extraction recipe on the fly. That long tail is our specialty. ### What does this Actor do? You paste a store URL. The Actor detects how the store is built, reads its catalog the fastest way that store allows (native JSON endpoints where they exist, sitemaps and product pages otherwise), and returns one clean JSON object per product — with **every variant** broken out as structured data. No API keys, no store login, no per-platform setup. - 🌍 **Works on any store, not just the famous ones** — Shopify, WooCommerce, PrestaShop and Magento are detected automatically; custom builds via AI. - 🧩 **Variants done properly** — sizes, colours and options as structured attributes, each with its own price, sale price, stock and image. Not just the parent product. - ⚡ **Fast where it can be** — Shopify catalogs read from JSON endpoints (a 1,000-product store in a handful of requests instead of a thousand page loads). - 💶 **Honest data** — transparent stock signals, real sale prices, no invented numbers. ### What data can you extract? | Product data | Commercial info | Per-variant detail | |---|---|---| | 📝 Name (multi-language) | 💰 Price & currency | 🎨 Attributes (size, colour, …) | | 🔗 Description | 🏷️ Sale price (discounted) | 💰 Variant price & sale price | | 🏢 Brand | 📂 Categories & tags | 📦 Stock status (in/out) | | 🖼️ All images (with position) | 🔗 Product & origin URL | 🔢 Stock quantity (when published) | | 🆔 Origin ID / SKU | 🧬 Product type (simple/variant) | 🖼️ Variant image | Every product is one JSON object; export the dataset as **JSON, CSV, Excel or XML**. ```json { "product": { "origin_id": "7811232948456", "origin_url": "https://the-store.com/products/example-sneaker", "name": {"en": "Example Sneaker"}, "brand": "ExampleBrand", "categories": ["Shoes", "Sneakers"], "base_price": 89.95, "base_discounted_price": 69.95, "currency": "EUR", "product_type": "variant" }, "variants": [ { "origin_variant_id": "43210", "sku": "SNKR-42-BLK", "variant_name": "42 / Black", "attributes": {"Size": "42", "Color": "Black"}, "price": 89.95, "discounted_price": 69.95, "stock_status": "in_stock", "stock_quantity": 7, "image_url": "https://..." } ], "images": [{"source_url": "https://...", "position": 0}] } ```` ### Which stores work? - **Shopify, WooCommerce, PrestaShop, Magento** (~70% of all shops) — detected automatically and scraped with a battle-tested template. Fast and cheap. - **Custom-built stores** (no recognisable platform) — turn on **"Scrape custom-built stores (AI)"**. An AI agent explores the site, figures out where the products, prices and variants live, and builds a recipe on the fly. Hard-capped per store so cost never runs away. Know the platform already? Set it in the **Platform** field to skip auto-detection. ### Why use this Actor? | | Typical single-platform scraper | Any E-commerce Store Catalog Scraper | |---|---|---| | Platform coverage | One platform, big sites only | Any store — 4 platforms + custom via AI | | Setup | API keys, auth, docs | Just paste a URL | | Variants | Limited or extra calls | Captured automatically, structured | | Small / regional shops | Usually unsupported | First-class — the long tail is the point | | Custom-built stores | Not supported | AI agent builds a recipe on the fly | | Cost | Per-request + subscription | Pay-per-event, only for what you get | ### What can you do with the data? - 💰 **Price & assortment monitoring** — schedule runs to track a competitor's catalog, prices and sale events over time. Pipe changes to Slack or a database. - 🛒 **Catalog imports & dropshipping** — pull a supplier's full catalog with variants ready to load into your own store. - 📊 **Market research** — map a niche across dozens of small stores nobody else covers, in one place. - 🗄️ **Migration & backup** — export your own store's catalog as portable JSON. ### How to use it 1. Create a free **Apify** account. 2. Open this Actor and paste the **store URL**. 3. Leave **Platform** on auto-detect (or pick it if you know it). 4. Optionally cap **Max products** to control cost. 5. For non-platform shops, turn on **Scrape custom-built stores (AI)**. 6. Click **Start** and export the dataset as JSON / CSV / Excel. ### How much does it cost? Pay-per-event — you're charged for what you actually get: | What | Price | |---|---| | Each product extracted | **$0.001** ($1 / 1,000 products) | | Custom-store AI onboarding (once per custom store) | **$2 flat** | - **Platform stores** (Shopify / WooCommerce / PrestaShop / Magento) pay only the per-product price. A 500-product store ≈ **$0.50**. - **Custom stores** add the one-off **$2** AI onboarding fee (the AI reads the site once and builds a recipe). A 500-product custom store ≈ **$2.50**. That flat fee covers the AI cost regardless of catalog size, so a tiny custom shop never runs at a loss. **Bring your own Anthropic API key** to run the AI on your own account — then you skip the onboarding fee and pay only per product. **Cost tips:** cap **Max products** while testing; set the **Platform** field to skip detection; leave the AI option off unless you actually need custom stores. ### Fair use & compliance - Respects `robots.txt` by default (opt-out only for stores you own or have permission to scrape). - Only **public catalog data** is extracted — no personal data, no accounts, no login walls. - Polite rate limiting with automatic backoff, so stores are never hammered. ### Support Open an issue on this Actor — we usually answer within 24h. If a specific store fails, include the URL: store-specific recipe fixes ship fast. # Actor input Schema ## `storeUrl` (type: `string`): The e-commerce store to scrape, e.g. https://my-shop.com. The platform (Shopify, WooCommerce, PrestaShop, Magento) is detected automatically. ## `platform` (type: `string`): Skip auto-detection if you already know the platform. ## `maxProducts` (type: `integer`): Stop after this many products (cost control). Leave empty for the full catalog. ## `stockDetail` (type: `string`): Shopify only: recover exact stock quantities via one extra request per product. 'Auto' probes a few products first and skips the slow pass when the shop doesn't publish quantities. Availability (in/out of stock) is ALWAYS included regardless. ## `customStoreLlm` (type: `boolean`): For stores that are NOT Shopify/WooCommerce/PrestaShop/Magento, let an AI agent build an extraction recipe by exploring the site. Costs more (see pricing) and is capped per store. Leave off to only scrape the four supported platforms. ## `llmBudgetUsd` (type: `string`): Hard ceiling on AI spend for each custom store. The run stops cleanly if reached. Only applies when 'Scrape custom-built stores' is on. ## `anthropicApiKey` (type: `string`): Optional: bring your own Anthropic key so the AI runs on YOUR account (cheaper per-result pricing applies). Leave empty to use ours (custom-store pricing applies). ## `respectRobotsTxt` (type: `boolean`): Skip URLs disallowed by the store's robots.txt. Only disable this when the store owner has authorised the scrape. ## Actor input object example ```json { "storeUrl": "https://", "platform": "auto", "stockDetail": "auto", "customStoreLlm": false, "llmBudgetUsd": "2", "respectRobotsTxt": true } ``` # Actor output Schema ## `products` (type: `string`): All extracted products with their variants and images. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "storeUrl": "https://" }; // Run the Actor and wait for it to finish const run = await client.actor("itsyas/store-catalog-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "storeUrl": "https://" } # Run the Actor and wait for it to finish run = client.actor("itsyas/store-catalog-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "storeUrl": "https://" }' | apify call itsyas/store-catalog-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=itsyas/store-catalog-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "E-commerce Store Catalog Scraper", "description": "92% success. Give it any store URL — get the full product catalog as clean JSON: products, variants, prices, sale prices, stock and images. Auto-detects Shopify, WooCommerce, PrestaShop and Magento.", "version": "0.1", "x-build-id": "fH6X5RlxTYaK0VvE9" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/itsyas~store-catalog-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-itsyas-store-catalog-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/itsyas~store-catalog-scraper/runs": { "post": { "operationId": "runs-sync-itsyas-store-catalog-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/itsyas~store-catalog-scraper/run-sync": { "post": { "operationId": "run-sync-itsyas-store-catalog-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "storeUrl" ], "properties": { "storeUrl": { "title": "Store URL", "type": "string", "description": "The e-commerce store to scrape, e.g. https://my-shop.com. The platform (Shopify, WooCommerce, PrestaShop, Magento) is detected automatically." }, "platform": { "title": "Platform", "enum": [ "auto", "shopify", "woocommerce", "prestashop", "magento" ], "type": "string", "description": "Skip auto-detection if you already know the platform.", "default": "auto" }, "maxProducts": { "title": "Max products", "minimum": 1, "type": "integer", "description": "Stop after this many products (cost control). Leave empty for the full catalog." }, "stockDetail": { "title": "Numeric stock detail (Shopify)", "enum": [ "auto", "always", "never" ], "type": "string", "description": "Shopify only: recover exact stock quantities via one extra request per product. 'Auto' probes a few products first and skips the slow pass when the shop doesn't publish quantities. Availability (in/out of stock) is ALWAYS included regardless.", "default": "auto" }, "customStoreLlm": { "title": "Scrape custom-built stores (AI)", "type": "boolean", "description": "For stores that are NOT Shopify/WooCommerce/PrestaShop/Magento, let an AI agent build an extraction recipe by exploring the site. Costs more (see pricing) and is capped per store. Leave off to only scrape the four supported platforms.", "default": false }, "llmBudgetUsd": { "title": "AI budget cap per store (USD)", "enum": [ "1", "2", "3" ], "type": "string", "description": "Hard ceiling on AI spend for each custom store. The run stops cleanly if reached. Only applies when 'Scrape custom-built stores' is on.", "default": "2" }, "anthropicApiKey": { "title": "Your Anthropic API key (optional)", "type": "string", "description": "Optional: bring your own Anthropic key so the AI runs on YOUR account (cheaper per-result pricing applies). Leave empty to use ours (custom-store pricing applies)." }, "respectRobotsTxt": { "title": "Respect robots.txt", "type": "boolean", "description": "Skip URLs disallowed by the store's robots.txt. Only disable this when the store owner has authorised the scrape.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```