Scrape Sephora products across 20 storefronts (US, Canada, 9 EU markets, 10 APAC countries) through one unified Python actor. Extract prices, variants, ratings, and catalog details via official mobile APIs with TLS fingerprint impersonation, OAuth2/guest-token auth, and per-market session isolation.
23/08/2026 — v2.6.1 MENA product-search wire corrected (bug fix)
Two defects on /app/backend/v1/shopper/product-search, both of which failed silently — the gateway ignores unrecognised query params rather than rejecting them, so neither ever surfaced as an error.
Fix: category crawls returned the entire catalogue, mislabelled as one category
The actor sent categoryId, which this gateway ignores: a valid C342 returned all 15,827 products where the storefront shows 919.
The category needs two params sent together — refine=cgid=<id>andtargetCategoryId=<id>. Alone, targetCategoryId is ignored and refine answers 400 CatalogValidationError, which is why neither showed up in isolation. Both now sent; verified live: C342 → 918 products, and the top-level ids match the gateway's own advertised counts (C302 2,615 / C303 2,434 / C301 2,013 / C304 758 / C307 1,107).
Fix: pagination never advanced — every page re-fetched the first
The actor sent start/count; the gateway uses offset/limit, so every request came back at offset: 0 with the default page size. Proven live: offset=100&limit=5 returns offset: 100 with different products, start=100&count=5 returns offset: 0 with 25.
Provenance
Param names and the key=value refine encoding come from the AE app's Retrofit interface rne#b and its nid request model (com.sephora.sephorame, classes3.dex): b(@Query("refine") List<String>, @Query("q"), @Query("sort"), @Query("offset") long, @Query("limit") int, @Query("locale"), @Query("targetCategoryId")).
The cgid refinement in the search response advertises its own valid value set (top-level C301/C302/C303/C304/C307 plus new, bestsellers, gifts-me, …), which is how the pairing was found — sub-category ids like C342 also work once both params are present.
Tests: tests/test_mena_category.py pins the paired category params, the refine encoding and the offset/limit wire. Verified live 2026-08-23 (AE residential proxy): category C342: 918 products, 60 unique records across multiple pages, 0 failed.
Docs
categoryIds was documented as EU-only. It is read by EU, MENA, LATAM, UK and India — each in that market's own id format (SFCC C… ids for EU/MENA/LATAM, numeric ids for UK, category slugs for India). US and SEA ignore it; use category URLs there.
The onlyNewProducts notes gave the wrong reason for MENA's exclusion. A usable mechanism does now exist (the search catalog's new category, ~281 products); it simply is not wired up.
Follow-up available: that new category means MENA can be added to onlyNewProducts — not done here to keep this a pure bug fix. In the meantime it can be crawled directly via categoryIds.
19/08/2026 — v2.6 Incremental refresh: only fetch recently-added products
Feat: onlyNewProducts — skip the catalogue, fetch what's new (19 of 28 markets)
Re-scraping a whole storefront to catch a handful of changes is slow and expensive. Setting onlyNewProducts restricts every category listing to recently-added products: on the US catalogue that is 1,201 products in ~21 listing requests, against 10,784 products / ~180 requests for a full sweep (measured 2026-08-19).
What "new" means depends on what each storefront's API can actually do, and the run logs which of the two applies:
Mechanism
Markets
Result
Filter
US/CA + EU (FR, IT, DE, ES, PL, RO, PT)
Exactly the products Sephora flags as new
Sort
SEA (all 10) + IN
The most recently published, capped by newProductsPageLimit
US/CA send ref=filters[isNew]=true on the v2 catalog endpoint (skincare 2,971 → 383).
EU sends refine_5=c_tagsEnabled=<value> on SFCC product_search. All seven storefronts mark new products with the same internal tag (tag6, established by correlating c_productFlags[].name against each market's localised label), but they do not accept the same refinement value: DE accepts only the raw id tag6enable (its labels return 0), ES/IT/PL/PT only the localised label (the raw id returns 0), FR/RO either. The value is therefore a per-market constant (EU_NEW_TAG) resolved locally, with no discovery round-trip. Verified live on C342 (baseline → filtered): fr 886→113, it 778→101, de 776→91, es 704→107, pl 763→99, pt 704→107, ro 844→120.
SEA and IN cannot filter, only sort, so they use their storefront's own newest-first ordering — ?sort=published_at on SEA's category listing and sort_on=latest on IN's Fynd endpoint. Because a sort returns the whole category merely reordered, both are bounded by the new newProductsPageLimit input (default 3 ≈ 70-110 products per category).
MENA and UK are refused, not silently widened.NEW_PRODUCTS_SUPPORT in src/markets/__init__.py is the single source of truth; markets absent from it are skipped with an ERROR rather than returning their full catalogue at the customer's cost. UK exposes no newness anywhere — filters are price/brand/badge/discount_level (badge = Bundle/Free Gifts/Savings), sorts are rating/price/discount (checked on a 319-result query). MENA advertises a c_tag6enable "New in" refinement (927 of 15,731) and a new-arrival sort id, but refine=, refine_1=, bare params and sort= all fail to apply it.
The filter is carried onto every page of a paginated crawl, and sort markets carry their page cap the same way — dropping either after page 1 would silently widen the run back to the whole category.
Feat: advancedFilter — raw refinement escape hatch (US/CA)
Sends any US refinement expression as ref, e.g. filters[shoppingPreferences]=koreanBeauty. Valid expressions are discoverable in the refinements block of any category response.
A ref= already present on a start URL is honoured too and wins over both inputs; advancedFilter wins over onlyNewProducts. Whenever a lower-precedence source is overridden it is logged, so a filter never vanishes silently.
Both failure modes are silent, and in opposite directions
US with the wrong spelling returns the full set: the expression must be sent as ref=filters[isNew]=true; a bare filters[isNew]=true is accepted and then ignored, with no error. The builder and its tests pin the working form.
EU with a wrong tag value returns total=0, also with no error. The EU search handler therefore warns when a filtered page comes back empty, so a stale tag cannot masquerade as "no new products".
Support was decided per market by applying each mechanism live and watching the count move, never from metadata — a category that does not advertise c_tagsEnabled in its refinements can still be filtered by it (RO C342: unadvertised, still 844→120).
Unchanged wire for existing runs
With no filter set, the US category URL is byte-identical to v2.5 — the frozen v1.x wire is preserved, and a regression test pins it.
Tests: tests/test_only_new_products.py (15 cases) covers the support matrix, the page cap, precedence, request building, the exact upstream URL per market, pagination carry-through, and the frozen unfiltered wire. Sample input at test-new-products.json.
Verified live (2026-08-19, residential proxy): a six-category US incremental run produced 46 records / 46 unique / 0 failed / no URL leaks, and filtered vs unfiltered /shop/skincare return different product sets. EU on C342: eu-fr 8 records and eu-de 13 records (EUR, no URL leaks), exercising both the localised-label and raw-id forms. SEA and IN with newProductsPageLimit=1: sea-nz 21 records (NZD) and in 9 records (INR), both logging the page cap and stopping. MENA and UK URLs produce 0 records and a clear per-market error.
Known issue found during this work, not fixed here
MENA category listing is broken, independently of this feature. The categoryId parameter is ignored entirely — a valid C342 returns 15,731 products (the whole catalogue) where the storefront shows 919 — and _parse_url rejects both live MENA category URL shapes (/shop/{cat}/{sub}/{ID} and the /c/ form). Every MENA category crawl therefore pulls the full catalogue. The correct parameter is still unknown; the AE app's Retrofit interface is the place to look.
18/08/2026 — v2.5.3 US /beauty/ editorial pages supported (bug fix)
Fix: www.sephora.com/beauty/{slug} URLs were rejected as "Invalid / Unsupported url"
Root cause: the US market recognised exactly two start-URL shapes — /shop/{slug} (category) and /product/…-P{id} (detail). /beauty/{slug} matched neither, so the reported URL https://www.sephora.com/beauty/new-skin-care-products?icid2=action_link_new_skincare was dropped before enqueue and the run exited with nothing to scrape.
Why it isn't just a wider regex:/beauty/ pages are not catalog categories. The storefront's own soft links tag them "type": "PageContent" (vs "PageCategory" for /shop/), and the v2 categories endpoint answers their slugs with 404 There is no matching category for :/categories/new-skin-care-products/seo. They are Contentful-backed curated grids served by a separate v1 content endpoint, recovered from the Retrofit interface wu0.b#getBeautyContentPage in com.sephora 26.6 (@GET("content/beauty/{slug}") + @Query("ch") + @Query("loc"), no countryCode) and confirmed live.
New content request kind:src/masker.py gains a content kind, src/auth/us_urls.py builds the real URL with the app's exact param names and order, and a new us-content handler reads data.layout.skuList and enqueues one detail request per product. The other four layout types (buying guide, customer services, directory, open) hold editorial copy, not products — they are skipped with a market-tagged log line rather than failing the run.
Known gap — this emits the page's editorial list, which is probably not what a visitor sees.layout.skuList (9 items for new-skin-care-products) is an SSR/editorial list. The page's visible grid is a separate Constructor.io-ranked feed at https://www.sephora.com/gway/v1/dotcom-sys/browse/{layout.ctorPodId} — same collection (display_name "New Skincare", pod id supplied by the page itself, and the only product call the page makes) but a substantially different set: only 1 of the 9 pinned ids appears in the feed's first 100, and the feed is broader (lip balms, value sets, a hand sanitizer). It is not "the rest of the page" — the two lists barely intersect. Verified 2026-08-19 by rendering the page: the visible grid is the feed (first tile P480529 Touchland Power Mist Hand Sanitizer, then Supergoop! PLAY, Vacation Mini Classic Whip, rhode Glazing Milk, rhode Highlight Milk — the feed's order, none of them in skuList), so what this actor emits for a /beauty/ URL is 9 products a visitor does not see there.
Why it is not wired up yet (investigated 2026-08-18). It is reachable — earlier notes here blamed Kasada, then Akamai; both were wrong, and the TCP drops were self-inflicted by probing with the app User-Agent over a Chrome JA3. The feed returns HTTP 200 to plain curl_cffi with the market's ordinary Seph-Access-Token. What stops it being a drop-in win: it caps at 200 results per sweep at any page size and reorders between requests (~160-180 unique of 383; 216 across two sweeps), and it returns nothing for the bestsellers pods on these parameters. So switching to it is a product decision — a different, larger, non-deterministic set — not a bug fix. For a guaranteed-complete assortment, use the matching /shop/… category URL, which paginates fully.
Canada:sephora.com/ca/en/beauty/… and /ca/fr/… resolve through the same path and produce CAD records with /ca/en canonical URLs.
Output hygiene:source.crawlUrl on every record stays the public /beauty/ page the customer passed (tracking params intact); the v1 content endpoint never reaches the dataset or the logs.
Scope note: only /beauty/{slug} is claimed. A multi-segment /beauty/{a}/{b} (e.g. /beauty/guides/how-to-apply-blush) is still reported unsupported — the app declares the slug as an @Path, which percent-encodes slashes, so that is not a shape the upstream API can serve.
Tests: URL classification (incl. the reported URL verbatim, both CA locales, trailing slash, multi-segment rejection), /shop/ and PDP regression cases, synthetic-request routing, the exact upstream URL, and handler behaviour across grid / non-grid / empty / 404 pages are locked in (tests/test_us_content_pages.py, 15 cases). The e2e manifest schema gained a content URL shape, wired so the operator runner picks it up automatically.
Verified live (2026-08-18, US residential proxy): the reported URL produced 9 records, 0 failed requests, every crawlUrl the public /beauty/ page and every canonicalUrl a public /product/ page. CA (/ca/en/beauty/…, CA proxy): 3 records, CAD, 0 failed.
11/08/2026 — v2.5.2 UK slug PDP URLs resolved (bug fix)
Fix: www.sephora.co.uk/p/{slug} product URLs failed with G-2 "could not be found" (HTTP 400)
Resolution hop: a /p/{slug} URL is now detected by token shape (P\d{4,} = id, else slug) and routed through a new resolve request that queries /catalog/v1/products?query={slug-as-text} (the only search param the gateway honors — q/term/keyword return a static suggestion set) and matches the slug against each result's normalized {brand} {name} (exact / prefix / ≥0.6 token overlap). The winning P… id is enqueued as a normal detail request. No confident match → recorded not-found, never a wrong product.
Output fix (source.canonicalUrl): the record now carries the real public /p/{slug} page the request started from, not a /p/{productId} deeplink or the internal fetch endpoint.
Proxy: UK reads only clear Akamai from a GB residential IP (datacenter and reputation-flagged residential exits get 403). A market may now declare preferred_proxy_country; when every active market agrees and the customer left the country unset, the run defaults to it (UK → GB). Mixed runs and an explicit proxy.apifyProxyCountry are untouched.
Latency: the Akamai-blocked /login/v1/login burst is no longer re-run on every catalog request — after one failed attempt it backs off for 5 min (reads don't need the session), saving ~6s + three dead round-trips per call.
Tests: slug-vs-id parsing, slug→id matching (incl. the reported URL), request routing, canonicalUrl threading, and the proxy-country pin are locked in (tests/test_uk_resolution.py, 21 cases).
Verified live (2026-08-11, residential proxy, country auto-pinned to GB): the reported URL https://www.sephora.co.uk/p/rhode-pocket-blush-buildable-hydrating-cream-blush resolved to P1000209341 and produced a valid record (title, 12 variants, 5 images, GBP, canonical = the public slug URL) — 2 requests, 0 failed.
07/08/2026 — v2.5.1 MENA storefront URLs accepted (bug fix)
Fix: www.sephora.me product URLs were rejected as "Invalid / Unsupported url"
Root cause: the six MENA markets were registered under per-country hostnames (www.sephora.ae, www.sephora.sa, …) inferred during Phase 1 recon and never verified live. The real consumer storefront is the single host www.sephora.me with a /{cc}-{lang}/ path prefix — the AE app's own declared production web base. Unregistered hostnames fall back to the us market, which then rejected every MENA URL and tagged the error [us].
Routing:detect_market now resolves www.sephora.me (and the apex) by path prefix — /ae-en/, /sa-en/, … → mena-ae, mena-sa, …. Bare (/ae/) and Arabic (/ae-ar/) prefixes resolve to the same market. An absent or unknown prefix defaults to AE rather than falling through to us. The legacy per-country hostnames stay registered.
URL parsing: the live grammar joins the product id with a slash (/p/{slug}/P{id}); the parser only accepted the hyphen form ({slug}-P{id}). Both now parse. ?productVariantId= is ignored — every variant is already emitted.
Output fix (source.canonicalUrl): MENA records carried a wrong host and a doubled slash (https://www.sephora.ae/ae//mega-mix-palette/P10061352). They now emit the real page URL, e.g. https://www.sephora.me/ae-en/p/mega-mix-palette/P10061352. This changes existing MENA output — consumers keying on the old value must update.
Diagnosability: an unrecognized hostname now logs a warning naming the host and the fallback market, once per host, instead of failing silently under the wrong market's tag.
All families: apex hostnames now resolve both directions. Previously only www.x → x was tried, so e.g. https://sephora.fr/... fell through to us; it now resolves to eu-fr. Same bug class as the MENA fault above.
Tests: the five reported URLs are locked in as routing + parse regressions; all six MENA path prefixes are covered; the canonicalUrl assertion is now exact rather than a prefix check (the prefix check is what let the malformed URL ship). README market table and the AE e2e manifest updated to the real grammar.
Verified live (2026-08-07, residential proxy, country-pinned): all 6 MENA markets PASS — 28 records, 0 failed requests, every canonicalUrl on the real host with the correct locale prefix. Run: tests/e2e/runs/mena-20260807T141315Z.jsonl.
Fix: live e2e gate was red for every market since v2.5 (unrelated pre-existing bug)
tests/e2e/scripts/assert_dataset.py kept its own hardcoded copy of the record's top-level key list. v2.5 added size to dataset_schema.json, all seven market transforms, and test_dataset_schema_compat.py — but not to this copy, so the script rejected every record with unexpected keys: ['size']. Any Layer-3 run since 30/06/2026 would have failed on all markets, not just MENA.
The script now derives the key set from .actor/dataset_schema.json instead of duplicating it, so the schema stays the single source of truth and this drift cannot recur.
NEWoptions[].values[].description — per-shade undertone/finish text, mapped from the US API's variationDesc (e.g. "light, neutral peach", "with neutral undertones", "medium warm 4"). Colour shades only; blank for size/scent variations.
NEW top-level size — headline size/volume of the default SKU (US: currentSku.size, e.g. "1 oz / 30 mL"). Non-US markets emit "" (their SFCC/GraphQL APIs don't expose a clean product-level size).
NEWvariants[].size — per-SKU size/volume (US-only; ancillary minis differ from the headline size).
Scope: undertone/finish + size are US/CA only — the international storefronts (SFCC EU/LATAM/MENA, GraphQL UK/SEA/IN) carry no per-shade description, so description is absent and size is "" there.
Schema docs updated: dataset_schema.json, test_dataset_schema_compat.py, and API_DOCUMENTATION.md.
09/05/2026 — v2.4 Phase 4 IN ship + Phase 3 UK reconciliation
Feat: Phase 4 — IN (Sephora India on Fynd Platform)
Added in family — single market on api.sephora.cn's Reliance-owned cousin api.fynd.com. Static Bearer auth (Base64 of compile-time applicationId:applicationToken recovered from com.jio.sephora 1.0.34). Path grammar /service/application/catalog/v1.0/products/{slug}/. Slug-based product addressing, page_no/page_size pagination. Module name in_market (avoids Python in keyword); family slug "in" in _FAMILY map.
Added latam-mx (es-MX, MXN) and latam-br (pt-BR, BRL) on apps.sephora.eu with /latam/ URL prefix and OCAPI v23_2 (newer than EU's v18_3). Single SODA key + DW client ID across both countries; per-country UA via LATAM_VERSIONS_BY_COUNTRY. Recovered from latam.sephora.sephoramx 26.4.3 + latam.sephora.sephorabr 26.4.2.
Feat: Phase 3 — UK (initial scaffold)
Added uk family — single market on apps.sephora.eu with a flatter REST surface (/catalog/v1/, /login/v1/, /content/v1/) than EU's OCAPI commerceCloud paths. Recovered from co.uk.sephora 19.7.1 (React Native + Hermes + Akamai BMP). x-consumer-id: AOC-ANDROID-UK (vs EU's AOC-ANDROID-EME).
08/05/2026 — v2.2 Phase 1 MENA ship
Feat: Phase 1 — MENA (6 countries on SCAPI/Apigee)
Added mena-ae, mena-sa, mena-bh, mena-om, mena-kw, mena-qa on apps.sephora.me (Apigee gateway + SCAPI). Per-country routing via lowercase apikey header carrying per-country Apigee key (AE + SA have their own; BH/OM/KW/QA fall back to AE's, mirroring how the App Store routes them). Six countries served by one backend. Recovered from ae.sephora.sephorauae 3.45.0-5386 + sa.sephora.sephorasa 3.45.0-5385.
18/04/2026 — v2.1 Crawlee migration + URL masking
Unified Crawlee HttpCrawler across US + EU + SEA with synthetic-URL masking, global circuit breaker, per-market pre-flight, declarative retry policy, serialized EU guest-token refresh, EU product-ID case normalization, maxProducts cap honored in EU pagination; CZ, GR, and PPE instrumentation removed, PT now served via the Spain backend with pt-PT locale.
NEWvariants[].wishlisted boolean field (SEA-only; analog of US lovesCount).
Feat:
Multi-market support — US + Canada + 9 EU countries + 10 APAC countries in one actor.
Auto-detect market from startUrls hostname; no input changes required for existing US users.
Optional market and locale override inputs for edge cases (bare IDs, region mismatches).
Per-market observability counters persisted to the default KV store as run-summary.
Architecture:
Refactored src/main.py into a lightweight dispatcher. Per-region logic lives in src/markets/{us,eu,sea}.py.
EU and SEA ports use curl_cffi for upstream compatibility. US keeps Crawlee's HttpCrawler with the existing session-pool auth flow unchanged.
Strict market-module isolation — per-region auth state cannot cross-contaminate.
Compatibility:
Existing US run configs work unchanged. Pre-v2.0 output fields are preserved; only the additive market key is new.
Standalone autofacts/sephora-eu-scraper and autofacts/sephora-nz-scraper listings remain live but are pinned to their current version. New bug fixes and features land in this repo only.
Deps:
Added: curl_cffi[async] >= 0.7, lxml >= 5.0.
10/03/2026
Feat: add sentiments field to product output, merging sentiment summary and sentiment items
