# E-Commerce Catalog Intelligence (Shopify, WooCommerce) (`piotrhl/ecommerce-catalog-scraper`) Actor

Fast, ultra-lightweight e-commerce catalog extractor and AI/RAG data pipeline. Extracts clean products, variants, and Markdown summary cards from Shopify, WooCommerce, and Schema.org JSON-LD stores under 128 MB RAM.

- **URL**: https://apify.com/piotrhl/ecommerce-catalog-scraper.md
- **Developed by:** [Piotr Hładik](https://apify.com/piotrhl) (community)
- **Categories:** E-commerce, AI
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## E-Commerce Catalog Intelligence (Shopify, WooCommerce & Schema.org JSON-LD)

[![Apify Actor](https://img.shields.io/badge/Apify-Actor-orange.svg)](https://apify.com)
[![Monetization](https://img.shields.io/badge/Monetization-Pay--Per--Event-brightgreen.svg)](https://apify.com)
[![RAM Footprint](https://img.shields.io/badge/Memory-%3C128MB-blue.svg)](https://apify.com)
[![Python 3.11+](https://img.shields.io/badge/Python-3.11+-blue.svg)](https://python.org)
[![Engine](https://img.shields.io/badge/Engine-HTTP%2F2%20Fast%20Core-teal.svg)](https://httpx.automattic.com)

**E-Commerce Catalog Intelligence** is an enterprise-grade, ultra-lightweight web scraper and AI data pipeline built for growth engineers, competitive intelligence analysts, and AI application builders. It extracts clean, normalized product catalogs from almost any modern e-commerce storefront—including **Shopify**, **WooCommerce**, and any web store exposing **Schema.org `application/ld+json`** structured data.

***

### ⚡ Core Features

- 🚀 **Zero Browser Bloat (Sub-128 MB RAM)**: Unlike legacy scrapers that waste money on headless Chromium (Playwright/Puppeteer) taking 1GB+ RAM, this Actor runs on an asynchronous HTTP/2 core with C-accelerated HTML parsing (`selectolax.lexbor`). It consumes under 128 MB RAM, maximizing your margins.
- 🔄 **Triple-Engine Auto-Detection**: Intelligently probes endpoints in sequence (`Shopify products.json` $\rightarrow$ `WooCommerce Store API v1` $\rightarrow$ `Schema.org JSON-LD Fallback`) or allows you to force a specific engine.
- 🏷️ **Accurate Currency & Minor Unit Conversion**: Correctly handles WooCommerce minor-unit integer prices (e.g., converting `2499` with minor unit 2 into `$24.99` / `€24.99`).
- 🤖 **AI & RAG Ready**: Automatically compiles an AI-optimized Markdown summary card (`llm_markdown_summary`) for every single product, making catalog ingestion into vector databases and LLM agents instant.
- 🛡️ **Pay-Per-Event (PPE) Cost Control**: Built-in budget protection that detects client charge limits and terminates gracefully without corrupting runs or wasting credits.

***

### 🔌 B2B Integrations: n8n, Make.com, Clay & RAG

#### 1. Clay Lead Enrichment & Account Scoring

Enrich e-commerce store leads with full catalog telemetry:

1. In Clay, add an **HTTP API / Apify integration action**.
2. Pass the company domain (`https://{{domain}}`) to `start_urls`.
3. Extract total products, price range (`price_min` to `price_max`), top brands, and variant counts straight into your CRM or enriched table.

#### 2. Make.com Automated Sync & Price Tracking

Automate catalog syncs, stock alerts, and competitor monitoring:

1. Add the **Apify "Run Actor and Get Dataset Items"** module.
2. Provide your target store URL.
3. Route output records to Google Sheets, Airtable, Notion, or Slack notifications for price drops.

#### 3. n8n Autonomous Agent Pipelines

Feed live e-commerce data into autonomous workflows:

1. Schedule cron runs in n8n targeting competitor domains.
2. Ingest `llm_markdown_summary` directly into vector stores (Qdrant, Pinecone, Chroma).
3. Query the knowledge base with LangChain or Claude to answer shopper questions or compile market research reports.

***

### ⚙️ Input Configuration

The Actor features an intuitive dual-mode interface: a **Standard Mode** for one-click extraction and an **Advanced / Nerd Mode** for network and engine fine-tuning.

#### Standard Mode

| Field | Type | Default | Description |
|---|---|---|---|
| `start_urls` | Array | `["https://fentybeauty.com"]` | List of store homepages, collection pages, or product URLs. |
| `max_products_per_store` | Integer | `100` | Max products to extract per store (up to 10,000). |
| `generate_llm_markdown` | Boolean | `true` | Generate pre-formatted Markdown summary cards for direct LLM/RAG ingestion. |

#### Advanced / Nerd Mode (Engine & Network Tuning)

| Field | Type | Default | Description |
|---|---|---|---|
| `concurrency` | Integer | `5` | Number of concurrent network workers (1 to 25). |
| `request_timeout_sec` | Integer | `20` | Timeout in seconds for HTTP requests. |
| `force_engine` | String | `"auto"` | Choose from `"auto"`, `"shopify"`, `"woocommerce"`, or `"jsonld"`. |
| `custom_user_agent` | String | `None` | Optional custom HTTP User-Agent string. |
| `include_raw_payload` | Boolean | `false` | Include raw JSON/HTML payload for schema debugging. |
| `custom_headers` | Object | `{}` | Key-value pairs of custom HTTP headers. |
| `proxy_configuration` | Object | `{"useApifyProxy": true}` | Apify Proxy settings to bypass geo-restrictions and rate limits. |

***

### 📊 Normalized Output Schema

Each extracted item is delivered in a structured, validated format:

```json
{
  "source_url": "https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer",
  "platform": "shopify",
  "product_id": "819283012",
  "title": "Gloss Bomb Universal Lip Luminizer",
  "brand": "Fenty Beauty",
  "description_clean": "An explosive lip gloss that feels as good as it looks. One luscious swipe gives lips more to love.",
  "price_min": 21.0,
  "price_max": 21.0,
  "currency": "USD",
  "in_stock": true,
  "images": [
    "https://cdn.shopify.com/s/files/1/gloss_bomb.jpg"
  ],
  "variants": [
    {
      "sku": "FB-GB-01",
      "title": "Fenty Glow",
      "price": 21.0,
      "currency": "USD",
      "in_stock": true,
      "url": "https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer?variant=11111"
    },
    {
      "sku": "FB-GB-02",
      "title": "Fu$$y",
      "price": 21.0,
      "currency": "USD",
      "in_stock": false,
      "url": "https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer?variant=22222"
    }
  ],
  "llm_markdown_summary": "### Gloss Bomb Universal Lip Luminizer\n- **Brand**: Fenty Beauty\n- **Price**: USD 21.00\n- **Availability**: In Stock\n- **Options**: 2 variant(s) [Fenty Glow, Fu$$y]\n- **Summary**: An explosive lip gloss that feels as good as it looks.\n- **Source**: [Gloss Bomb Universal Lip Luminizer](https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer)",
  "scraped_at": "2026-10-17T19:54:30.123456+00:00"
}
```

***

### 💰 Pay-Per-Event (PPE) Pricing Transparency

This Actor operates under Apify's **Pay-Per-Event (PPE)** pricing model. You only pay for what you extract, eliminating idle compute costs:

| Event | Price (USD) | When Charged |
|---|---|---|
| `apify-actor-start` | **$0.00005** | Charged once at container initialization. |
| `product-extracted` | **$0.002** | Charged per product extracted with complete variants and AI Markdown. |

#### Why PPE Saves You Money

- Traditional compute-based scrapers charge for every second a browser is spinning, even when waiting on slow servers.
- With PPE, if a catalog has 50 items, you pay exactly for 50 items ($$0.10$).
- If a target site is offline or returns 0 products, you don't burn dollars on idle compute.

***

### ❓ Frequently Asked Questions (FAQ)

#### Does this Actor require residential proxies?

For standard public Shopify and WooCommerce stores, datacenter proxies or direct requests work well. For stores protected by Cloudflare or strict geo-restrictions, enable Apify Residential Proxies in the `proxy_configuration` input field.

#### How does the fallback mechanism work?

When set to `auto` (default), the Actor first probes `/products.json?limit=1`. If that succeeds, it runs the ultra-fast Shopify engine. If not, it probes `/wp-json/wc/store/v1/products?per_page=1` for WooCommerce. If both return 404, it downloads the HTML document and extracts Schema.org `application/ld+json` (`@graph` and `Product` microdata).

#### Can I scrape individual product pages instead of whole stores?

Yes. You can supply direct product URLs (e.g., `https://store.com/products/my-item` or `https://shop.com/product/hoodie`). The Actor automatically extracts just that product.

#### How do I use the output with LangChain or OpenAI?

Simply query the Apify dataset and pass the `llm_markdown_summary` string as the `page_content` in your LangChain `Document` objects. It is pre-formatted with clean markdown headers and bullet points.

***

### 🛠️ Local Development & Testing

```bash
## Clone the repository
git clone https://github.com/your-org/ecommerce-catalog-intelligence.git
cd ecommerce-catalog-intelligence

## Install dependencies into your local virtual environment
.venv\Scripts\python.exe -m pip install -r requirements.txt

## Run full test suite with 100% mocked offline requests
.venv\Scripts\python.exe -m pytest -v

## Run locally using Apify CLI
apify run
```

***

### 📜 License & Compliance

Distributed under the Apache 2.0 License. Ensure scraping activities comply with store terms of service and relevant local privacy regulations.

# Actor input Schema

## `start_urls` (type: `array`):

List of e-commerce store homepages, collection pages, or product URLs to extract.

## `max_products_per_store` (type: `integer`):

Maximum number of products to extract from each store or catalog.

## `generate_llm_markdown` (type: `boolean`):

Produce an AI-ready Markdown summary card for each product, optimized for RAG embeddings and LLM pipelines.

## `concurrency` (type: `integer`):

Number of parallel workers processing stores or product pages concurrently.

## `request_timeout_sec` (type: `integer`):

HTTP request timeout in seconds.

## `force_engine` (type: `string`):

Force a specific extraction engine or let the actor auto-detect the e-commerce platform.

## `custom_user_agent` (type: `string`):

Custom HTTP User-Agent header string.

## `include_raw_payload` (type: `boolean`):

Include the unparsed raw JSON or HTML schema payload in output items for debugging.

## `custom_headers` (type: `object`):

Optional custom HTTP headers to pass with every outbound request.

## `proxy_configuration` (type: `object`):

Apify Proxy settings to bypass rate limits or geoblocking.

## Actor input object example

```json
{
  "start_urls": [
    {
      "url": "https://fentybeauty.com"
    }
  ],
  "max_products_per_store": 100,
  "generate_llm_markdown": true,
  "concurrency": 5,
  "request_timeout_sec": 20,
  "force_engine": "auto",
  "include_raw_payload": false,
  "custom_headers": {},
  "proxy_configuration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "start_urls": [
        {
            "url": "https://fentybeauty.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("piotrhl/ecommerce-catalog-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "start_urls": [{ "url": "https://fentybeauty.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("piotrhl/ecommerce-catalog-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "start_urls": [
    {
      "url": "https://fentybeauty.com"
    }
  ]
}' |
apify call piotrhl/ecommerce-catalog-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,piotrhl/ecommerce-catalog-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RYEh8r38fD7lmifCQ/builds/raggGzcVXYwIrqLBs/openapi.json