# U Ecommerce Scraper (`kalirobot/u-ecommerce-scraper`) Actor

An intelligent, universal e-commerce scraper for Apify that automatically detects 6+ store systems, finds product data via sitemaps or deep crawling, and normalizes it into a clean, structured format.

- **URL**: https://apify.com/kalirobot/u-ecommerce-scraper.md
- **Developed by:** [Dennis](https://apify.com/kalirobot) (community)
- **Categories:** E-commerce
- **Stats:** 280 total users, 4 monthly users, 100.0% runs succeeded, 12 bookmarks
- **User rating**: 1.43 out of 5 stars

## Pricing

$3.00 / 1,000 products

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Universal E-Commerce Scraper

[![Apify Actor](https://img.shields.io/badge/Apify-Actor-blue.svg)](https://apify.com)
[![Python](https://img.shields.io/badge/Python-3.9+-yellow.svg)](https://www.python.org/downloads/)
[![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://www.docker.com/)

**An intelligent, universal e-commerce scraper for Apify that automatically detects 6+ store systems, finds product data via sitemaps or deep crawling, and normalizes it into a clean, structured format.**

***

### Core Features

| Feature                 | Description                                                                                             | Status      |
| ----------------------- | -------------------------------------------------------------------------------------------------------- | ----------- |
| **Automatic Platform Detection** | Detects **Shopify, WooCommerce, Magento, PrestaShop, BigCommerce, Shopware**, and generic systems.    | ✅ **Active** |
| **Intelligent Product Discovery**  | **1. Sitemap Analysis:** Finds product URLs via `sitemap.xml` (fast & efficient).                    | ✅ **Active** |
|                         | **2. Deep Product Crawler:** "Screaming Frog"-style crawling for intelligent product page identification. | ✅ **Active** |
| **Robust Data Extraction**      | Uses APIs (WooCommerce, Shopify), GraphQL, JSON-LD, and HTML parsing for maximum accuracy.      | ✅ **Active** |
| **Data Normalization**        | Cleans and standardizes prices, currencies, availability, and URLs for consistent output.        | ✅ **Active** |
| **Ethical Crawling**          | Optional respect for `robots.txt`, `Crawl-Delay`, and bot identification.                               | ✅ **Active** |
| **Proxy & User-Agent Management** | Full Apify Proxy integration and User-Agent rotation to prevent blocking.          | ✅ **Active** |
| **Anti-Marketplace Protection**    | A blacklist prevents crawling large marketplaces like Amazon, eBay, etc.                         | ✅ **Active** |
| **Flexible Output Formats**     | Exports data as **JSON, CSV, or XLSX**.                                                            | ✅ **Active** |

***

### 🚫 Marketplace Policy

This scraper is designed **exclusively for standalone online stores**. It is **NOT** suitable for large marketplaces such as **Amazon, eBay, Alibaba, Otto, Kaufland, Walmart, Etsy**, etc. Attempts to crawl these sites will be actively blocked.

***

### 🚀 Quick Start on the Apify Platform

1. **Click "Try actor".**
2. **Enter your start URLs** (e.g., `https://your-shopify-store.com/`, `https://your-magento-store.com/`).
3. **Configure the options** (see "Input Parameters").
4. **Click "Start"** to begin the crawl.

You will find the results in the **Dataset tab** once the run is complete.

***

### ⚙️ Input Parameters

For a full description of all input parameters, please refer to the **[Input Schema Tab](/.actor/input_schema.json)**.

| Key Parameters        | Description                                                                                                                              |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `startUrls`               | The URLs of the stores you want to crawl.                                                                                             |
| `maxProducts`             | The maximum number of products to extract. Ideal for quick tests.                                                                      |
| `enableDeepCrawler`       | **(Warning: Can be expensive!)** Activates the intelligent deep crawler if sitemap analysis fails.                                     |
| `proxyConfiguration`      | Configure Apify Proxy or your own custom proxies. **(Recommended)**                                                                       |
| `outputFormat`            | Choose the output format: `json`, `csv`, or `xlsx`.                                                                                    |
| `includeDescription`, `includeImages`, `includeSKU` | Specify which optional fields should be included in the results.                                            |
| `respectRobotsTxt`        | Ensures the crawler respects the website's `robots.txt` rules.                                                          |
| `rotateUserAgents`        | Rotates user agents for each request to avoid getting blocked.                                                                             |
| `debugMode`, `saveHTML`   | Enable these options for detailed debugging. Saves raw HTML to the Key-Value Store.                                         |

***

### 📊 Example Output (JSON)

The extracted data is provided in a clean, standardized format.

```json
[
{
    "name": "Stylish T-Shirt - Blue",
    "description": "A comfortable and stylish t-shirt made from 100% organic cotton. Perfect for everyday wear.",
  "price": 29.99,
    "currency": "USD",
    "sku": "TS-BL-M-01",
  "availability": "InStock",
    "image": "https://your-store.com/images/stylish-t-shirt-blue.jpg",
    "url": "https://your-store.com/products/stylish-t-shirt-blue",
  "platform": "shopify",
    "variants": [
      {
        "name": "Color",
        "value": "Blue"
      },
      {
        "name": "Size",
        "value": "M"
      }
    ]
  }
]
```

*Note: The `attributes` field (for detailed specifications) is primarily supported for WooCommerce and Magento stores.*

# Actor input Schema

## `startUrls` (type: `array`):

List of URLs where the crawl should start.

## `maxCrawlDepth` (type: `integer`):

Maximum depth for crawling from the start URLs (0 = only start URLs, 1 = start URLs + 1 level, etc.).

## `maxCrawlPages` (type: `integer`):

Maximum number of pages to crawl.

## `maxProducts` (type: `integer`):

Maximum number of products to extract (leave empty for unlimited).

## `detectShopPlatform` (type: `boolean`):

Automatically detect platforms like Magento, Shopify, WooCommerce, etc.

## `respectRobotsTxt` (type: `boolean`):

Ethical Crawling: Obeys robots.txt disallow rules and crawl delays. Recommended for legal/ethical operation.

## `identifyAsBot` (type: `boolean`):

Uses an identifiable User-Agent (ApifyPriceScraperBot). If false, uses a standard browser User-Agent.

## `rotateUserAgents` (type: `boolean`):

Uses random browser User-Agents for each request (anti-bot evasion). Overrides 'Identify as a Bot'.

## `crawlDelay` (type: `number`):

Minimum time between requests to the same domain. 0 = no delay.

## `enableDeepCrawler` (type: `boolean`):

⚠️ WARNING: Crawls ALL links on the site (Screaming Frog-style). Can lead to high costs! Only enable if sitemap is missing.

## `includeUrlPatterns` (type: `array`):

Only crawl URLs that match these regex patterns (empty = all URLs).

## `excludeUrlPatterns` (type: `array`):

Skip URLs that match these regex patterns.

## `minPrice` (type: `number`):

Only extract products with prices above this value (optional).

## `maxPrice` (type: `number`):

Only extract products with prices below this value (optional).

## `onlyInStock` (type: `boolean`):

Only extract products that are currently in stock.

## `sortBy` (type: `string`):

Sort extracted products by this field.

## `productType` (type: `string`):

Filter by WooCommerce product type (only effective for WooCommerce stores).

## `proxyConfiguration` (type: `object`):

Proxy settings for the crawler (recommended for larger stores).

## `maxConcurrency` (type: `integer`):

Number of simultaneous HTTP requests (higher = faster, but higher risk of blocking).

## `pageLoadTimeout` (type: `integer`):

Maximum time to wait for a page to load (in seconds).

## `customUserAgent` (type: `string`):

Custom User-Agent string (overrides default and rotation).

## `cookies` (type: `array`):

Custom cookies for authentication or personalization.

## `customHeaders` (type: `object`):

Additional HTTP headers for all requests.

## `outputFormat` (type: `string`):

Format of the extracted data in the dataset.

## `includeImages` (type: `boolean`):

Include product images in the output.

## `includeDescription` (type: `boolean`):

Include product descriptions in the output.

## `includeSKU` (type: `boolean`):

Include product SKUs in the output.

## `includeAvailability` (type: `boolean`):

Include stock status in the output.

## `debugMode` (type: `boolean`):

Enables verbose logging for debugging (shows extraction details).

## `saveHTML` (type: `boolean`):

Saves the HTML code of each product page in the Key-Value Store.

## `notifyUrl` (type: `string`):

Push notification upon completion. Supports: ntfy.sh URLs or Pushover API.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 3,
  "maxCrawlPages": 100,
  "detectShopPlatform": true,
  "respectRobotsTxt": false,
  "identifyAsBot": false,
  "rotateUserAgents": false,
  "crawlDelay": 0,
  "enableDeepCrawler": false,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [
    "\\.(pdf|jpg|jpeg|png|gif|css|js|svg|ico|woff|woff2|ttf)$"
  ],
  "onlyInStock": false,
  "sortBy": "none",
  "productType": "any",
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxConcurrency": 1,
  "pageLoadTimeout": 60,
  "customUserAgent": "MyCustomBot/1.0",
  "cookies": [],
  "customHeaders": {
    "X-Custom-Header": "value"
  },
  "outputFormat": "json",
  "includeImages": true,
  "includeDescription": true,
  "includeSKU": true,
  "includeAvailability": true,
  "debugMode": false,
  "saveHTML": false,
  "notifyUrl": "https://ntfy.sh/your-topic"
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `allItems` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ],
    "customHeaders": {}
};

// Run the Actor and wait for it to finish
const run = await client.actor("kalirobot/u-ecommerce-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://example.com" }],
    "customHeaders": {},
}

# Run the Actor and wait for it to finish
run = client.actor("kalirobot/u-ecommerce-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "customHeaders": {}
}' |
apify call kalirobot/u-ecommerce-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kalirobot/u-ecommerce-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/g76ZE3C0d1HKkGS7e/builds/EbNn5XhKsoJK0Xzmc/openapi.json