# Shopee Scraper -  Products & Suppliers, One for All (`scrapenode/shopee-scraper-products-suppliers-one-for-all`) Actor

Search Shopee products over pure HTTP — no browser, no cookie, no login — and turn the results into a supplier list you can compare. Built for price research and supplier discovery in Indonesia.

- **URL**: https://apify.com/scrapenode/shopee-scraper-products-suppliers-one-for-all.md
- **Developed by:** [Muhammad Sajid Izzulhaq](https://apify.com/scrapenode) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Shopee Scraper — Products & Suppliers, One for All

Search products on Shopee, then group their sellers into a comparable supplier list. You only supply product keywords and optional filters.

The actor uses a **single pure-HTTP data path** — our own client, no browser, no cookie, no login, and no third-party vendor.

### What it does

Shopee locks its JSON search API (`GET /api/v4/search/search_items`) behind an anti-bot gate: a plain HTTP client gets HTTP 200 with `{"error": 90309999}`, and the product page's server-rendered data has price and sold count stripped out (`removed_fields`).

But Shopee still serves a **fully server-rendered search page to social crawlers**. Ask for the search page with a social-crawler User-Agent (`WhatsApp/2.23.20.0`) and the response is ~230 KB of HTML containing **40 complete product cards** — name, price, strikethrough price, discount, rating, and location — with no cookie, no token, and no browser. (The `facebookexternalhit` UA that used to work is blocked with HTTP 403 as of ~Oct 2026.)

So the actor:

1. Requests `https://<market-domain>/search?keyword=...` with the `WhatsApp`-family User-Agent (the default market `ID` uses `https://shopee.co.id/search`).
2. Parses each `aria-label="Product card"` block into a product row.
3. Deduplicates by `itemId`, applies optional filters, and aggregates products into shops.
4. Writes the dataset and the `products.xlsx` spreadsheet.

### Markets

The actor supports seven Shopee markets, selected with the `country` input. **One country per run.** The default is `ID`, so existing runs are unchanged.

| `country` | Domain | Currency | Symbol | Sample price format |
|---|---|---|---|---|
| `ID` | shopee.co.id | IDR | Rp | `1.299.000` |
| `BR` | shopee.com.br | BRL | R$ | `2.999,01` |
| `SG` | shopee.sg | SGD | S$ | `1,234.56` |
| `MY` | shopee.com.my | MYR | RM | `1,650.00` |
| `VN` | shopee.vn | VND | ₫ | `1.899.000` |
| `TH` | shopee.co.th | THB | ฿ | `12,490` |
| `PH` | shopee.ph | PHP | ₱ | `24,899` |

The market also sets the `Accept-Language` header, the product/shop URL host, the price parsing (thousands/decimal separators), and the **proxy egress country** — the residential proxy is requested in the selected country's code, so prices and street names match the market. The `currency` of each product is written to the dataset (see `Currency` column).

All seven markets were verified live (HTTP 200 + 40 cards each) with the same card structure; see `data/scraped/_debug/shopee-multimarket-probe.json`.

#### Fields that are unavailable over this path

The server-rendered card does not carry two fields, so they are left **empty rather than guessed**:

- **`shopName`** — the card has no shop name (the product page redacts it too). The shop **id** is still there (`shopId`), so `shopUrl` and shop aggregation still work.
- **`countSold`** — the sold count is not rendered on the card.

`ratingCount`, `likedCount`, `categoryId`, `isAd`, and `shopeeVerified` are likewise not on the card and are `null`/`false`.

### Architecture

- **Transport & parsing** live in `shopee_core.html_search` (shared package): one GET per page with rate limiting and exponential backoff, then a regex parser per product card.
- **Aggregation & reporting** live in `shopee_core.records` and `shopee_core.reports` (shared with the actor's siblings).

### Input

| Input | Type | Default | Description |
|---|---|---|---|
| `searchStrings` | list of string | `["laptop"]` | One or more product keywords. All are combined in a single run so the **Shops** sheet sees every seller that appears under any keyword. |
| `country` | enum | `"ID"` | Shopee market: `ID`, `BR`, `SG`, `MY`, `VN`, `TH`, `PH`. Sets the domain, currency, price locale, and proxy country. One market per run. |
| `maxItems` | integer | `100` | Limit of unique products **for the whole run**. Maximum `600`. |
| `sortBy` | enum | `"relevancy"` | `relevancy` or `sales`. |
| `skipAds` | boolean | `false` | Currently inactive: the server-rendered card does not mark ads. |
| `proxy` | proxy | Apify `RESIDENTIAL`, country matching `country` (`ID` default) | Apify residential proxy by default; replace with your own if needed. |
| `minPrice` | number | — | Drop products below this price (in the market's currency). |
| `maxPrice` | number | — | Drop products above this price (in the market's currency). |
| `minRating` | number | — | Drop products rated below this (0–5). Products without a rating are dropped too. |
| `debugDump` | boolean | `false` | Save the raw HTML pages to storage. |

#### Proxy

Three paths, tried in order: the **`proxy` input component** (default Apify `RESIDENTIAL`, country matching the selected market), the **`SCRAPER_PROXY`** env var, then the **Apify residential proxy** built from `APIFY_PROXY_PASSWORD` (egress country = selected market). Credentials are never written to code or logs (they are redacted).

### Output

One dataset row = one product, 22 columns:

`no`, `query`, `page`, `itemId`, `name`, `url`, `price`, `priceOriginal`, `discountPercent`, `currency`, `countSold`, `rating`, `ratingCount`, `likedCount`, `shopId`, `shopName`, `shopUrl`, `shopLocation`, `shopeeVerified`, `categoryId`, `isAd`, `imageUrl`

Two views in the Output tab: **Products** (reading order when comparing suppliers) and **All columns** (every field, raw).

Two artifacts land in the key-value store:

- **`products.xlsx`** — three sheets: `Products`, `Shops` (aggregated per seller), and `Summary` (run config + stats).
- **`SUMMARY`** — run statistics, including `rawItems`, `duplicates`, `filtered`, and `shops`.

The **`Products`** sheet shows only the columns that are actually filled over the HTML path (`No`, `Keyword`, `Image`, `Product Name`, `Price`, `Original Price`, `Discount`, `Rating`, `Location`, `Page`, `Product URL`, `Shop on Shopee`). The dead columns (`Sold`, `Reviews`, `Ad`, `Shop`, `Verified`, `Category`) are omitted so the table stays clean — their raw values still live in the dataset. The **`Image`** cell is a hyperlink that opens the product image.

#### Reading the columns

- **`price` is the full amount in the market's currency** parsed from the card (e.g. `42900` for ID, `2999.01` for BR), not the 1/100 000 internal unit the JSON API uses.
- **`currency`** is the ISO code of the market the run targeted (e.g. `IDR`, `SGD`). Prices are **not** converted; they are as shown on the market's own page.
- **`itemId` is the dedup key.** Shopee's index shifts between requests, so dedup runs across the whole result set.
- **`url`** is the product URL taken from the card (contains the slug, `shopId`, and `itemId`).
- **`rating` can be `null`** for products with no rating.
- **`imageUrl`** is the product's main image URL taken from the card (original resolution; no download is performed). In the `Products` sheet it becomes a clickable `View Image` cell.
- **`countSold` and `shopName` are always `null`** over this path (see above).

### Pagination and server limits

`maxItems` is a **total**, not per page. The number of cards per page is **not fixed**: it varies roughly between 10 and 60 depending on the keyword and when you run (page 1 can be as low as 10–20 for narrow keywords). The actor walks pages up to `MAX_PAGES_PER_QUERY` (10 pages per keyword) and stops when the target is reached or pages stop adding new products. Total results depend heavily on the keyword and are not stable between runs; use a broader keyword or several keyword variants for more results.

Shopee answers the search endpoint with **HTTP 403** when it blocks the egress IP, independent of cookies or sessions. The actor counts those pages as `blocked` (reported in `SUMMARY` as `blocked`/`statusCodes`) and, if a run ends with zero products because of it, **fails the run** with an explicit "IP proxy blocked" message instead of silently succeeding. In that case switch the proxy or re-run to get a different residential exit node.

### Known limitation — crawler User-Agent

This actor depends on the **server-rendered** search page Shopee serves to social-crawler User-Agents. That path is inherently fragile: Shopee periodically adds crawler UAs to a block list (for example, `facebookexternalhit` returned `200` until roughly Oct 2026, then started returning `403`). If Shopee blocks the User-Agent this actor currently uses, runs can fail even with a healthy proxy IP — rotating the proxy will not help, because the block is on the User-Agent, not the IP. This is a known operational risk; a mitigation is planned but not yet in place. If a run fails with a "rejected by the server" message and the proxy is confirmed working, it is likely this scenario.

### Running locally

```bash
mkdir -p .tmp-actor-run/storage/key_value_stores/default
## write input to .tmp-actor-run/storage/key_value_stores/default/INPUT.json
export SCRAPER_PROXY=http://user:pass@host:port          # optional locally
CRAWLEE_STORAGE_DIR=.tmp-actor-run/storage \
PYTHONPATH="actors/shopee-search;packages/shopee-core/src" \
  .venv/Scripts/python -m src
```

The actor's build context is the **repo root** (`dockerContextDir: "../../.."`), which is why `packages/shopee-core` must be copied along. Don't change it to `COPY . .`. The Docker base image is `apify/actor-python`; `curl_cffi` ships its own impersonation wheel, so the image stays small (~300 MB) — no Chromium.

### Tests

```bash
pytest packages/shopee-core/tests -q
```

Parsing regressions use a real server-rendered page in `data/scraped/_debug/shopee-headset-srp.html` as a fixture, and the HTTP transport is stubbed, so no internet connection is needed.

# Actor input Schema

## `searchStrings` (type: `array`):

One or more product keywords, e.g. 'laptop', 'running shoes', 'specialty coffee'. All keywords are combined in a single run so the Shops sheet sees every seller that appears under any keyword.

## `maxItems` (type: `integer`):

Limit of unique products for the whole run, split evenly across keywords. The server serves roughly 10-60 cards per page depending on the keyword; the actor paginates for you, so you don't need to think about per-page limits.

## `region` (type: `string`):

Pilih pasar target. Menentukan nilai `gl` dan proxy country yang digunakan oleh aktor. Pilihan: BR, ID, TH, MY, SG, PH, VN, MX.

## `sortBy` (type: `string`):

How Shopee orders the results. 'Relevancy' is the default; 'Best selling' is passed to the server as sortBy=sales.

## `skipAds` (type: `boolean`):

Currently inactive: the server-rendered HTML version does not mark ads, so there is nothing to drop. Kept for compatibility.

## `proxy` (type: `object`):

Shopee sometimes rejects datacenter IPs; residential proxies in the selected market's country are pre-selected as the default so you stay safe. Change only if you know what you're doing.

## `minPrice` (type: `number`):

Drop products priced below this amount, in the selected market's currency (no separators). Example: 50000 to drop products under IDR 50,000. Products without a price are dropped too.

## `maxPrice` (type: `number`):

Drop products priced above this amount, in the selected market's currency (no separators). Example: 500000 to drop products over IDR 500,000. Products without a price are dropped too.

## `minRating` (type: `number`):

Drop products rated below this value (0–5 scale). Example: 4.5. Products with no reviews (empty rating) are dropped too.

## `debugDump` (type: `boolean`):

Save the search responses as-is to storage.

## Actor input object example

```json
{
  "searchStrings": [
    "laptop"
  ],
  "maxItems": 100,
  "region": "ID",
  "sortBy": "relevancy",
  "skipAds": false,
  "proxy": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ],
    "apifyProxyCountry": "ID"
  },
  "minPrice": 50000,
  "maxPrice": 500000,
  "minRating": 4.5,
  "debugDump": false
}
```

# Actor output Schema

## `products` (type: `string`):

All Shopee search products, including price, currency, sold count, rating, ad flag, and the seller's data.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchStrings": [
        "laptop"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapenode/shopee-scraper-products-suppliers-one-for-all").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchStrings": ["laptop"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapenode/shopee-scraper-products-suppliers-one-for-all").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchStrings": [
    "laptop"
  ]
}' |
apify call scrapenode/shopee-scraper-products-suppliers-one-for-all --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapenode/shopee-scraper-products-suppliers-one-for-all"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zBfRFOOVG54g1cgLk/builds/CNpidyIf69P65LScU/openapi.json