Scrape Sephora products across 20 storefronts (US, Canada, 9 EU markets, 10 APAC countries) through one unified Python actor. Extract prices, variants, ratings, and catalog details via official mobile APIs with TLS fingerprint impersonation, OAuth2/guest-token auth, and per-market session isolation.
11/08/2026 — v2.5.2 UK slug PDP URLs resolved (bug fix)
Fix: www.sephora.co.uk/p/{slug} product URLs failed with G-2 "could not be found" (HTTP 400)
Resolution hop: a /p/{slug} URL is now detected by token shape (P\d{4,} = id, else slug) and routed through a new resolve request that queries /catalog/v1/products?query={slug-as-text} (the only search param the gateway honors — q/term/keyword return a static suggestion set) and matches the slug against each result's normalized {brand} {name} (exact / prefix / ≥0.6 token overlap). The winning P… id is enqueued as a normal detail request. No confident match → recorded not-found, never a wrong product.
Output fix (source.canonicalUrl): the record now carries the real public /p/{slug} page the request started from, not a /p/{productId} deeplink or the internal fetch endpoint.
Proxy: UK reads only clear Akamai from a GB residential IP (datacenter and reputation-flagged residential exits get 403). A market may now declare preferred_proxy_country; when every active market agrees and the customer left the country unset, the run defaults to it (UK → GB). Mixed runs and an explicit proxy.apifyProxyCountry are untouched.
Latency: the Akamai-blocked /login/v1/login burst is no longer re-run on every catalog request — after one failed attempt it backs off for 5 min (reads don't need the session), saving ~6s + three dead round-trips per call.
Tests: slug-vs-id parsing, slug→id matching (incl. the reported URL), request routing, canonicalUrl threading, and the proxy-country pin are locked in (tests/test_uk_resolution.py, 21 cases).
Verified live (2026-08-11, residential proxy, country auto-pinned to GB): the reported URL https://www.sephora.co.uk/p/rhode-pocket-blush-buildable-hydrating-cream-blush resolved to P1000209341 and produced a valid record (title, 12 variants, 5 images, GBP, canonical = the public slug URL) — 2 requests, 0 failed.
07/08/2026 — v2.5.1 MENA storefront URLs accepted (bug fix)
Fix: www.sephora.me product URLs were rejected as "Invalid / Unsupported url"
Root cause: the six MENA markets were registered under per-country hostnames (www.sephora.ae, www.sephora.sa, …) inferred during Phase 1 recon and never verified live. The real consumer storefront is the single host www.sephora.me with a /{cc}-{lang}/ path prefix — the AE app's own declared production web base. Unregistered hostnames fall back to the us market, which then rejected every MENA URL and tagged the error [us].
Routing:detect_market now resolves www.sephora.me (and the apex) by path prefix — /ae-en/, /sa-en/, … → mena-ae, mena-sa, …. Bare (/ae/) and Arabic (/ae-ar/) prefixes resolve to the same market. An absent or unknown prefix defaults to AE rather than falling through to us. The legacy per-country hostnames stay registered.
URL parsing: the live grammar joins the product id with a slash (/p/{slug}/P{id}); the parser only accepted the hyphen form ({slug}-P{id}). Both now parse. ?productVariantId= is ignored — every variant is already emitted.
Output fix (source.canonicalUrl): MENA records carried a wrong host and a doubled slash (https://www.sephora.ae/ae//mega-mix-palette/P10061352). They now emit the real page URL, e.g. https://www.sephora.me/ae-en/p/mega-mix-palette/P10061352. This changes existing MENA output — consumers keying on the old value must update.
Diagnosability: an unrecognized hostname now logs a warning naming the host and the fallback market, once per host, instead of failing silently under the wrong market's tag.
All families: apex hostnames now resolve both directions. Previously only www.x → x was tried, so e.g. https://sephora.fr/... fell through to us; it now resolves to eu-fr. Same bug class as the MENA fault above.
Tests: the five reported URLs are locked in as routing + parse regressions; all six MENA path prefixes are covered; the canonicalUrl assertion is now exact rather than a prefix check (the prefix check is what let the malformed URL ship). README market table and the AE e2e manifest updated to the real grammar.
Verified live (2026-08-07, residential proxy, country-pinned): all 6 MENA markets PASS — 28 records, 0 failed requests, every canonicalUrl on the real host with the correct locale prefix. Run: tests/e2e/runs/mena-20260807T141315Z.jsonl.
Fix: live e2e gate was red for every market since v2.5 (unrelated pre-existing bug)
tests/e2e/scripts/assert_dataset.py kept its own hardcoded copy of the record's top-level key list. v2.5 added size to dataset_schema.json, all seven market transforms, and test_dataset_schema_compat.py — but not to this copy, so the script rejected every record with unexpected keys: ['size']. Any Layer-3 run since 30/06/2026 would have failed on all markets, not just MENA.
The script now derives the key set from .actor/dataset_schema.json instead of duplicating it, so the schema stays the single source of truth and this drift cannot recur.
NEWoptions[].values[].description — per-shade undertone/finish text, mapped from the US API's variationDesc (e.g. "light, neutral peach", "with neutral undertones", "medium warm 4"). Colour shades only; blank for size/scent variations.
NEW top-level size — headline size/volume of the default SKU (US: currentSku.size, e.g. "1 oz / 30 mL"). Non-US markets emit "" (their SFCC/GraphQL APIs don't expose a clean product-level size).
NEWvariants[].size — per-SKU size/volume (US-only; ancillary minis differ from the headline size).
Scope: undertone/finish + size are US/CA only — the international storefronts (SFCC EU/LATAM/MENA, GraphQL UK/SEA/IN) carry no per-shade description, so description is absent and size is "" there.
Schema docs updated: dataset_schema.json, test_dataset_schema_compat.py, and API_DOCUMENTATION.md.
09/05/2026 — v2.4 Phase 4 IN ship + Phase 3 UK reconciliation
Feat: Phase 4 — IN (Sephora India on Fynd Platform)
Added in family — single market on api.sephora.cn's Reliance-owned cousin api.fynd.com. Static Bearer auth (Base64 of compile-time applicationId:applicationToken recovered from com.jio.sephora 1.0.34). Path grammar /service/application/catalog/v1.0/products/{slug}/. Slug-based product addressing, page_no/page_size pagination. Module name in_market (avoids Python in keyword); family slug "in" in _FAMILY map.
Added latam-mx (es-MX, MXN) and latam-br (pt-BR, BRL) on apps.sephora.eu with /latam/ URL prefix and OCAPI v23_2 (newer than EU's v18_3). Single SODA key + DW client ID across both countries; per-country UA via LATAM_VERSIONS_BY_COUNTRY. Recovered from latam.sephora.sephoramx 26.4.3 + latam.sephora.sephorabr 26.4.2.
Feat: Phase 3 — UK (initial scaffold)
Added uk family — single market on apps.sephora.eu with a flatter REST surface (/catalog/v1/, /login/v1/, /content/v1/) than EU's OCAPI commerceCloud paths. Recovered from co.uk.sephora 19.7.1 (React Native + Hermes + Akamai BMP). x-consumer-id: AOC-ANDROID-UK (vs EU's AOC-ANDROID-EME).
08/05/2026 — v2.2 Phase 1 MENA ship
Feat: Phase 1 — MENA (6 countries on SCAPI/Apigee)
Added mena-ae, mena-sa, mena-bh, mena-om, mena-kw, mena-qa on apps.sephora.me (Apigee gateway + SCAPI). Per-country routing via lowercase apikey header carrying per-country Apigee key (AE + SA have their own; BH/OM/KW/QA fall back to AE's, mirroring how the App Store routes them). Six countries served by one backend. Recovered from ae.sephora.sephorauae 3.45.0-5386 + sa.sephora.sephorasa 3.45.0-5385.
18/04/2026 — v2.1 Crawlee migration + URL masking
Unified Crawlee HttpCrawler across US + EU + SEA with synthetic-URL masking, global circuit breaker, per-market pre-flight, declarative retry policy, serialized EU guest-token refresh, EU product-ID case normalization, maxProducts cap honored in EU pagination; CZ, GR, and PPE instrumentation removed, PT now served via the Spain backend with pt-PT locale.
NEWvariants[].wishlisted boolean field (SEA-only; analog of US lovesCount).
Feat:
Multi-market support — US + Canada + 9 EU countries + 10 APAC countries in one actor.
Auto-detect market from startUrls hostname; no input changes required for existing US users.
Optional market and locale override inputs for edge cases (bare IDs, region mismatches).
Per-market observability counters persisted to the default KV store as run-summary.
Architecture:
Refactored src/main.py into a lightweight dispatcher. Per-region logic lives in src/markets/{us,eu,sea}.py.
EU and SEA ports use curl_cffi for upstream compatibility. US keeps Crawlee's HttpCrawler with the existing session-pool auth flow unchanged.
Strict market-module isolation — per-region auth state cannot cross-contaminate.
Compatibility:
Existing US run configs work unchanged. Pre-v2.0 output fields are preserved; only the additive market key is new.
Standalone autofacts/sephora-eu-scraper and autofacts/sephora-nz-scraper listings remain live but are pinned to their current version. New bug fixes and features land in this repo only.
Deps:
Added: curl_cffi[async] >= 0.7, lxml >= 5.0.
10/03/2026
Feat: add sentiments field to product output, merging sentiment summary and sentiment items
