Shopee Scraper — Products & Suppliers, One for All

Search products on Shopee, then group their sellers into a comparable supplier list. You only supply product keywords and optional filters.

The actor uses a single pure-HTTP data path — our own client, no browser, no cookie, no login, and no third-party vendor.

What it does

Shopee locks its JSON search API (GET /api/v4/search/search_items) behind an anti-bot gate: a plain HTTP client gets HTTP 200 with {"error": 90309999}, and the product page's server-rendered data has price and sold count stripped out (removed_fields).

But Shopee still serves a fully server-rendered search page to social crawlers. Ask for the search page with a social-crawler User-Agent (WhatsApp/2.23.20.0) and the response is ~230 KB of HTML containing 40 complete product cards — name, price, strikethrough price, discount, rating, and location — with no cookie, no token, and no browser. (The facebookexternalhit UA that used to work is blocked with HTTP 403 as of ~Oct 2026.)

So the actor:

1. Requests https://<market-domain>/search?keyword=... with the WhatsApp-family User-Agent (the default market ID uses https://shopee.co.id/search).
2. Parses each aria-label="Product card" block into a product row.
3. Deduplicates by itemId, applies optional filters, and aggregates products into shops.
4. Writes the dataset and the products.xlsx spreadsheet.

Markets

The actor supports seven Shopee markets, selected with the country input. One country per run. The default is ID, so existing runs are unchanged.

country	Domain	Currency	Symbol	Sample price format
ID	shopee.co.id	IDR	Rp	1.299.000
BR	shopee.com.br	BRL	R$	2.999,01
SG	shopee.sg	SGD	S$	1,234.56
MY	shopee.com.my	MYR	RM	1,650.00
VN	shopee.vn	VND	₫	1.899.000
TH	shopee.co.th	THB	฿	12,490
PH	shopee.ph	PHP	₱	24,899

The market also sets the Accept-Language header, the product/shop URL host, the price parsing (thousands/decimal separators), and the proxy egress country — the residential proxy is requested in the selected country's code, so prices and street names match the market. The currency of each product is written to the dataset (see Currency column).

All seven markets were verified live (HTTP 200 + 40 cards each) with the same card structure; see data/scraped/_debug/shopee-multimarket-probe.json.

Fields that are unavailable over this path

The server-rendered card does not carry two fields, so they are left empty rather than guessed:

• shopName — the card has no shop name (the product page redacts it too). The shop id is still there (shopId), so shopUrl and shop aggregation still work.
• countSold — the sold count is not rendered on the card.

ratingCount, likedCount, categoryId, isAd, and shopeeVerified are likewise not on the card and are null/false.

Architecture

• Transport & parsing live in shopee_core.html_search (shared package): one GET per page with rate limiting and exponential backoff, then a regex parser per product card.
• Aggregation & reporting live in shopee_core.records and shopee_core.reports (shared with the actor's siblings).

Input

Input	Type	Default	Description
searchStrings	list of string	["laptop"]	One or more product keywords. All are combined in a single run so the Shops sheet sees every seller that appears under any keyword.
country	enum	"ID"	Shopee market: ID, BR, SG, MY, VN, TH, PH. Sets the domain, currency, price locale, and proxy country. One market per run.
maxItems	integer	100	Limit of unique products for the whole run. Maximum 600.
sortBy	enum	"relevancy"	relevancy or sales.
skipAds	boolean	false	Currently inactive: the server-rendered card does not mark ads.
proxy	proxy	Apify RESIDENTIAL, country matching country (ID default)	Apify residential proxy by default; replace with your own if needed.
minPrice	number	—	Drop products below this price (in the market's currency).
maxPrice	number	—	Drop products above this price (in the market's currency).
minRating	number	—	Drop products rated below this (0–5). Products without a rating are dropped too.
debugDump	boolean	false	Save the raw HTML pages to storage.

Proxy

Three paths, tried in order: the proxy input component (default Apify RESIDENTIAL, country matching the selected market), the SCRAPER_PROXY env var, then the Apify residential proxy built from APIFY_PROXY_PASSWORD (egress country = selected market). Credentials are never written to code or logs (they are redacted).

Output

One dataset row = one product, 22 columns:

no, query, page, itemId, name, url, price, priceOriginal, discountPercent, currency, countSold, rating, ratingCount, likedCount, shopId, shopName, shopUrl, shopLocation, shopeeVerified, categoryId, isAd, imageUrl

Two views in the Output tab: Products (reading order when comparing suppliers) and All columns (every field, raw).

Two artifacts land in the key-value store:

• products.xlsx — three sheets: Products, Shops (aggregated per seller), and Summary (run config + stats).
• SUMMARY — run statistics, including rawItems, duplicates, filtered, and shops.

The Products sheet shows only the columns that are actually filled over the HTML path (No, Keyword, Image, Product Name, Price, Original Price, Discount, Rating, Location, Page, Product URL, Shop on Shopee). The dead columns (Sold, Reviews, Ad, Shop, Verified, Category) are omitted so the table stays clean — their raw values still live in the dataset. The Image cell is a hyperlink that opens the product image.

Reading the columns

• price is the full amount in the market's currency parsed from the card (e.g. 42900 for ID, 2999.01 for BR), not the 1/100 000 internal unit the JSON API uses.
• currency is the ISO code of the market the run targeted (e.g. IDR, SGD). Prices are not converted; they are as shown on the market's own page.
• itemId is the dedup key. Shopee's index shifts between requests, so dedup runs across the whole result set.
• url is the product URL taken from the card (contains the slug, shopId, and itemId).
• rating can be null for products with no rating.
• imageUrl is the product's main image URL taken from the card (original resolution; no download is performed). In the Products sheet it becomes a clickable View Image cell.
• countSold and shopName are always null over this path (see above).

Pagination and server limits

maxItems is a total, not per page. The number of cards per page is not fixed: it varies roughly between 10 and 60 depending on the keyword and when you run (page 1 can be as low as 10–20 for narrow keywords). The actor walks pages up to MAX_PAGES_PER_QUERY (10 pages per keyword) and stops when the target is reached or pages stop adding new products. Total results depend heavily on the keyword and are not stable between runs; use a broader keyword or several keyword variants for more results.

Shopee answers the search endpoint with HTTP 403 when it blocks the egress IP, independent of cookies or sessions. The actor counts those pages as blocked (reported in SUMMARY as blocked/statusCodes) and, if a run ends with zero products because of it, fails the run with an explicit "IP proxy blocked" message instead of silently succeeding. In that case switch the proxy or re-run to get a different residential exit node.

Known limitation — crawler User-Agent

This actor depends on the server-rendered search page Shopee serves to social-crawler User-Agents. That path is inherently fragile: Shopee periodically adds crawler UAs to a block list (for example, facebookexternalhit returned 200 until roughly Oct 2026, then started returning 403). If Shopee blocks the User-Agent this actor currently uses, runs can fail even with a healthy proxy IP — rotating the proxy will not help, because the block is on the User-Agent, not the IP. This is a known operational risk; a mitigation is planned but not yet in place. If a run fails with a "rejected by the server" message and the proxy is confirmed working, it is likely this scenario.

Running locally

mkdir -p .tmp-actor-run/storage/key_value_stores/default
# write input to .tmp-actor-run/storage/key_value_stores/default/INPUT.json
export SCRAPER_PROXY=http://user:pass@host:port          # optional locally
CRAWLEE_STORAGE_DIR=.tmp-actor-run/storage \
PYTHONPATH="actors/shopee-search;packages/shopee-core/src" \
  .venv/Scripts/python -m src

The actor's build context is the repo root (dockerContextDir: "../../.."), which is why packages/shopee-core must be copied along. Don't change it to COPY . .. The Docker base image is apify/actor-python; curl_cffi ships its own impersonation wheel, so the image stays small (~300 MB) — no Chromium.

Tests

$pytest packages/shopee-core/tests -q

Parsing regressions use a real server-rendered page in data/scraped/_debug/shopee-headset-srp.html as a fixture, and the HTTP transport is stubbed, so no internet connection is needed.