# Changelog of Shein Product Scraper — Prices, Variants & Reviews (`khadinakbar/shein-product-scraper`) Actor

- **URL**: https://apify.com/khadinakbar/shein-product-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/khadinakbar/shein-product-scraper.md

Tranche 2 cost-efficiency: defaultMemoryMbytes 2048 -> 1024 (2026-09-29).

### 0.7.5 — 2026-08-21

- Fixed managed Google public-index query: `inurl:-p- "phrase"` returned no product pages on Google. Now uses `site:{host} "/-p-" {keywords}` with a `site:shein.com` retry, accepts any `*.shein.com` host, and rewrites canonical URLs to the storefront host. Restores listing-only quality-probe rows when Shein GeeTest blocks the browser path.

### 0.7.4 — 2026-07-29

- Added a strict managed Google public-index path for listing-only requests. Tiny quality-test probes use genuine Google-indexed canonical Shein product pages first, while larger/richer jobs keep browser extraction as primary and use the index only after a zero-row armor failure. The fallback validates host + `-p-{goodsId}.html`, deduplicates goods IDs, preserves indexed price/rating/availability metadata, labels `scrapeSource=public-index`, charges only after a real dataset push, and never emits diagnostic rows. Added deterministic parser/provider tests.

### 0.7 — 2026-06-17

- Wired the full 2Captcha GeeTest-v4 solver (env TWO_CAPTCHA_API_KEY ← secret @twocaptcha_api_key). New src/geetest.js calls 2Captcha geetest_v4. An injected initGeetest4/initGeetest hook (page.addInitScript + page.exposeFunction \_\_sheinSolveGeetest) captures Shein's dynamically-fetched captcha_id at challenge time, solves it, and returns a fake captcha object whose getValidate() yields the solved {lot_number, captcha_output, pass_token, gen_time} + fires onSuccess so Shein runs its own /risk/verify/identity/validation submit and the page navigates back. Challenge wait extended to 85s when a solver is configured; logs window.\_\_SHEIN_GEE hook state on failure to confirm whether Shein calls the global initGeetest4. requestHandlerTimeout 90→150s for the solve.

### 0.6 — 2026-06-17

- Identified Shein's armor (captcha_type 909/903) as a GeeTest 'I am human' widget under Shein's /risk/verify/identity/validation flow (diagnosed via a DEBUG_DUMP_CHALLENGE build that captured the challenge HTML/screenshot/iframes to KV). Root cause of all prior product-page failures: the code only WAITED for an auto-redirect, which an interactive GeeTest never does. Added behavioral GeeTest handling — clickHumanVerify() clicks the radar/checkbox (main doc + iframes, humanized delay) and polls up to 22s for the page to navigate back; a stealthed browser frequently passes GeeTest on click without a puzzle. 2Captcha GeeTest-v4 solving remains the next fallback for puzzle escalations. Diagnostic dump re-gated behind DEBUG_DUMP_CHALLENGE so normal runs aren't slowed.

### 0.5 — 2026-06-17

- Throughput + consistency tuning after v0.4 proved sticky IPs extract real products (full title/price/18 images) but ran slow: the listing hit Shein's per-IP rate limit (risk/action/limit) and one product kept getting captcha_type=909. Lighter warm-up (waitUntil:'commit' so only the Set-Cookie response is fetched, not the 1MB homepage), challenge wait cut 18s→9s so blocked sessions rotate to a fresh sticky IP sooner (a fraction of IPs get a free pass), maxConcurrency 2→4 across distinct sticky IPs, sameDomainDelay 2→3s to ease rate-limiting, retries 5→4, sessionRotations 10→12.

### 0.4 — 2026-06-17

- Fixed the product-page block. v0.3 canary proved DataImpulse+stealth PASSES the Shein search/listing page (extracted real product cards) but product detail pages re-challenged (captcha_type 909/903). Root cause: DataImpulse's rotating gateway (:823) gives a new exit IP per connection, so the armor pass-cookie issued during warm-up/listing was invalid on the next request's IP. Fix: switched to DataImpulse STICKY ports (10000-19999, one IP per port → one stable IP per Crawlee session) so the warm-up → listing → product chain shares an IP and the armor cookie stays valid — same technique that beat DataDome in monster-jobs-scraper. Also strip Shein's bot-ish tracking params (detailBusinessFrom/pageListType) from enqueued product URLs.

### 0.3 — 2026-06-17

- Switched primary proxy to the user's DataImpulse residential proxy (Apify secret @dataimpulse-proxy → env DATAIMPULSE_PROXY) after Apify Residential access was exhausted (GB/credit cap) mid-test. DataImpulse rotating gateway (gw.dataimpulse.com:823) gives a fresh exit IP per connection; username is country-pinned with \_\_cr.<cc> matched to the storefront. Input proxy default relaxed from RESIDENTIAL to auto so run-start validation passes on accounts without Apify residential. Apify residential remains a fallback when DATAIMPULSE_PROXY is unset.

### 0.2 — 2026-06-17

- Anti-bot escalation after v0.1 canary was blocked on all 10 residential rotations (Chromium + Crawlee fingerprints did not pass Shein armor). Added playwright-extra + stealth plugin (masks navigator.webdriver/automation tells), disabled Crawlee fingerprint injection to avoid double-patching, pinned a consistent desktop Chrome UA + anti-automation launch flags. risk/challenge is now WAITED OUT (silent armor auto-redirects back after JS) for up to 18s with mouse-nudges before a session is rotated, instead of throwing immediately. Added light human mouse movement before extraction.

### 0.1 — 2026-06-17

- Initial release. All-in-one Shein scraper: keyword search, category URLs, product URLs, and goods IDs auto-detected from input.
- PlaywrightCrawler (Chromium) on Apify Residential proxy with fingerprint + session pool. Shein's homegrown 'armor' risk engine blocks all raw/warmed HTTP and the BFF JSON API (403) — a real browser is required to compute the armor token, so the actor renders pages and passively captures Shein's own BFF responses (get_goods_detail_static_data_v2 / realtime / image) plus an in-page state + regex fallback.
- risk/challenge redirect detection rotates to a fresh residential session (separate maxSessionRotations budget) with exponential backoff.
- Pay-per-event: actor-start $0.00005, product-scraped $0.04, search-result $0.005.
- Run-lifecycle hardening: soft-fail on empty/invalid input (SUCCEEDED + terminal WARNING), honest-fail only when every session was blocked with zero items, safePushData wrapper around every dataset write, upfront cost-cap log + triple-guard charge counter, RUN_SUMMARY written to KV.