E-Commerce Catalog Intelligence (Shopify, WooCommerce & Schema.org JSON-LD)

E-Commerce Catalog Intelligence is an enterprise-grade, ultra-lightweight web scraper and AI data pipeline built for growth engineers, competitive intelligence analysts, and AI application builders. It extracts clean, normalized product catalogs from almost any modern e-commerce storefront—including Shopify, WooCommerce, and any web store exposing Schema.org application/ld+json structured data.

---

⚡ Core Features

• 🚀 Zero Browser Bloat (Sub-128 MB RAM): Unlike legacy scrapers that waste money on headless Chromium (Playwright/Puppeteer) taking 1GB+ RAM, this Actor runs on an asynchronous HTTP/2 core with C-accelerated HTML parsing (selectolax.lexbor). It consumes under 128 MB RAM, maximizing your margins.
• 🔄 Triple-Engine Auto-Detection: Intelligently probes endpoints in sequence (Shopify products.json $\rightarrow$ WooCommerce Store API v1 $\rightarrow$ Schema.org JSON-LD Fallback) or allows you to force a specific engine.
• 🏷️ Accurate Currency & Minor Unit Conversion: Correctly handles WooCommerce minor-unit integer prices (e.g., converting 2499 with minor unit 2 into $24.99 / €24.99).
• 🤖 AI & RAG Ready: Automatically compiles an AI-optimized Markdown summary card (llm_markdown_summary) for every single product, making catalog ingestion into vector databases and LLM agents instant.
• 🛡️ Pay-Per-Event (PPE) Cost Control: Built-in budget protection that detects client charge limits and terminates gracefully without corrupting runs or wasting credits.

---

🔌 B2B Integrations: n8n, Make.com, Clay & RAG

1. Clay Lead Enrichment & Account Scoring

Enrich e-commerce store leads with full catalog telemetry:

1. In Clay, add an HTTP API / Apify integration action.
2. Pass the company domain (https://{{domain}}) to start_urls.
3. Extract total products, price range (price_min to price_max), top brands, and variant counts straight into your CRM or enriched table.

2. Make.com Automated Sync & Price Tracking

Automate catalog syncs, stock alerts, and competitor monitoring:

1. Add the Apify "Run Actor and Get Dataset Items" module.
2. Provide your target store URL.
3. Route output records to Google Sheets, Airtable, Notion, or Slack notifications for price drops.

3. n8n Autonomous Agent Pipelines

Feed live e-commerce data into autonomous workflows:

1. Schedule cron runs in n8n targeting competitor domains.
2. Ingest llm_markdown_summary directly into vector stores (Qdrant, Pinecone, Chroma).
3. Query the knowledge base with LangChain or Claude to answer shopper questions or compile market research reports.

---

⚙️ Input Configuration

The Actor features an intuitive dual-mode interface: a Standard Mode for one-click extraction and an Advanced / Nerd Mode for network and engine fine-tuning.

Standard Mode

Field	Type	Default	Description
start_urls	Array	["https://fentybeauty.com"]	List of store homepages, collection pages, or product URLs.
max_products_per_store	Integer	100	Max products to extract per store (up to 10,000).
generate_llm_markdown	Boolean	true	Generate pre-formatted Markdown summary cards for direct LLM/RAG ingestion.

Advanced / Nerd Mode (Engine & Network Tuning)

Field	Type	Default	Description
concurrency	Integer	5	Number of concurrent network workers (1 to 25).
request_timeout_sec	Integer	20	Timeout in seconds for HTTP requests.
force_engine	String	"auto"	Choose from "auto", "shopify", "woocommerce", or "jsonld".
custom_user_agent	String	None	Optional custom HTTP User-Agent string.
include_raw_payload	Boolean	false	Include raw JSON/HTML payload for schema debugging.
custom_headers	Object	{}	Key-value pairs of custom HTTP headers.
proxy_configuration	Object	{"useApifyProxy": true}	Apify Proxy settings to bypass geo-restrictions and rate limits.

---

📊 Normalized Output Schema

Each extracted item is delivered in a structured, validated format:

{
  "source_url": "https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer",
  "platform": "shopify",
  "product_id": "819283012",
  "title": "Gloss Bomb Universal Lip Luminizer",
  "brand": "Fenty Beauty",
  "description_clean": "An explosive lip gloss that feels as good as it looks. One luscious swipe gives lips more to love.",
  "price_min": 21.0,
  "price_max": 21.0,
  "currency": "USD",
  "in_stock": true,
  "images": [
    "https://cdn.shopify.com/s/files/1/gloss_bomb.jpg"
  ],
  "variants": [
    {
      "sku": "FB-GB-01",
      "title": "Fenty Glow",
      "price": 21.0,
      "currency": "USD",
      "in_stock": true,
      "url": "https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer?variant=11111"
    },
    {
      "sku": "FB-GB-02",
      "title": "Fu$$y",
      "price": 21.0,
      "currency": "USD",
      "in_stock": false,
      "url": "https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer?variant=22222"
    }
  ],
  "llm_markdown_summary": "### Gloss Bomb Universal Lip Luminizer\n- **Brand**: Fenty Beauty\n- **Price**: USD 21.00\n- **Availability**: In Stock\n- **Options**: 2 variant(s) [Fenty Glow, Fu$$y]\n- **Summary**: An explosive lip gloss that feels as good as it looks.\n- **Source**: [Gloss Bomb Universal Lip Luminizer](https://fentybeauty.com/products/gloss-bomb-universal-lip-luminizer)",
  "scraped_at": "2026-10-17T19:54:30.123456+00:00"
}

---

💰 Pay-Per-Event (PPE) Pricing Transparency

This Actor operates under Apify's Pay-Per-Event (PPE) pricing model. You only pay for what you extract, eliminating idle compute costs:

Event	Price (USD)	When Charged
apify-actor-start	$0.00005	Charged once at container initialization.
product-extracted	$0.002	Charged per product extracted with complete variants and AI Markdown.

Why PPE Saves You Money

• Traditional compute-based scrapers charge for every second a browser is spinning, even when waiting on slow servers.
• With PPE, if a catalog has 50 items, you pay exactly for 50 items ($$0.10$).
• If a target site is offline or returns 0 products, you don't burn dollars on idle compute.

---

❓ Frequently Asked Questions (FAQ)

Does this Actor require residential proxies?

For standard public Shopify and WooCommerce stores, datacenter proxies or direct requests work well. For stores protected by Cloudflare or strict geo-restrictions, enable Apify Residential Proxies in the proxy_configuration input field.

How does the fallback mechanism work?

When set to auto (default), the Actor first probes /products.json?limit=1. If that succeeds, it runs the ultra-fast Shopify engine. If not, it probes /wp-json/wc/store/v1/products?per_page=1 for WooCommerce. If both return 404, it downloads the HTML document and extracts Schema.org application/ld+json (@graph and Product microdata).

Can I scrape individual product pages instead of whole stores?

Yes. You can supply direct product URLs (e.g., https://store.com/products/my-item or https://shop.com/product/hoodie). The Actor automatically extracts just that product.

How do I use the output with LangChain or OpenAI?

Simply query the Apify dataset and pass the llm_markdown_summary string as the page_content in your LangChain Document objects. It is pre-formatted with clean markdown headers and bullet points.

---

🛠️ Local Development & Testing

# Clone the repository
git clone https://github.com/your-org/ecommerce-catalog-intelligence.git
cd ecommerce-catalog-intelligence

# Install dependencies into your local virtual environment
.venv\Scripts\python.exe -m pip install -r requirements.txt

# Run full test suite with 100% mocked offline requests
.venv\Scripts\python.exe -m pytest -v

# Run locally using Apify CLI
apify run

---

📜 License & Compliance

Distributed under the Apache 2.0 License. Ensure scraping activities comply with store terms of service and relevant local privacy regulations.