Sephora product data from 29 countries in one run: US, Canada, 9 EU countries, 6 Middle East markets, the UK, India and 10 Asia-Pacific markets. Every variant with price, stock, shade, size, ingredients, rating and review count, in one schema. Paste product or category URLs. Pay per product saved.
Store title, SEO title, SEO description and description rewritten in buyer terms: what comes back, from which 29 countries, how it is billed. The description no longer describes the fetch mechanism.
One count everywhere: 29 countries (US, Canada, 9 EU, 6 Middle East, UK, India, 10 Asia-Pacific). README, actor.json, input schema and dataset schema each carried a different number before.
Review wording corrected: the actor returns rating and review count (plus US sentiment summaries), not review texts.
categoryIds help text in the input form now matches the README: read by EU, Middle East, UK and India. LATAM dropped from customer-facing text until that market is verified.
Dataset schema descriptions no longer name Sephora's upstream fields, and the overview view no longer claims integer minor-unit prices.
README field table covers every storefront family; the "Production-tested" bullet naming retired actors is gone. One actor name across Console, actor.json, input form and README.
httpx is now declared in requirements.txt. Builds from 16/09/2026 reported nothing to Scraper Insight — neither request logs nor the run summary.
Telemetry that disables itself now logs at WARNING, not INFO.
Run logs no longer name the telemetry endpoint or the app id.
15/09/2026 — v2.7.0 Revenue ledger line per run (feature)
The actor reported every upstream request to Scraper Insight but never reported what a run earned, so it was missing from the fleet earnings page entirely — despite being the second-highest earner.
Added: one ledger line per run
report_run_summary() in src/insight.py posts the run's charged-event counts and the per-event prices that applied to that caller's subscription tier to /api/v1/run-summary. Apify publishes no creator-earnings API — /v2/users/me/{insights,monetization,earnings,payouts} all return 404 — so this is the only record of what a run charged.
Neither billable event is charged by actor code: apify-actor-start is charged by the platform at run start and seeded into the SDK's ChargingManager when it initializes, and apify-default-dataset-item is mirrored into the charging state on every dataset write. Reading the manager therefore needed no change at any call site.
The run's business shape (catalog / product / mixed) is tallied from the labels of the requests actually enqueued, and its item count comes from the same per-market counters the final log summary reports. Raw request counts do not track revenue; catalog-type work does.
Sent as a single awaited request with its own 10-second budget, not through InsightReporter's queue — that queue is drained and closed by the time the run ends, and its 3-second budget is sized for fire-and-forget request logs.
Changed: main() split into a ledger-writing wrapper and _run()
The crawl had five exit paths and the ledger is only trustworthy if it has every run, so the summary is now written once in a finally, with the status derived from whether the crawl raised. The bare raise is kept so reporting cannot mask a crawl failure.
The two early exits that called await Actor.exit() now simply return. Exiting the Actor also shuts the ChargingManager down, which would leave the summary with nothing to read; the enclosing async with Actor: exits cleanly anyway.
Tests
tests/test_insight.py pins the wire format against the Go db.FleetRun decoder: field names, Decimal → float conversion (stdlib json has no Decimal encoder, so a stray one would raise inside the POST and lose the run's money silently), an uncapped maxTotalChargeUsd omitted rather than sent as a null the dashboard would read as a zero ceiling, a run still reported when the charging state is unreadable, and no report at all from a local run.
23/08/2026 — v2.6.1 MENA product-search wire corrected (bug fix)
Two defects on /app/backend/v1/shopper/product-search, both of which failed silently — the gateway ignores unrecognised query params rather than rejecting them, so neither ever surfaced as an error.
Fix: category crawls returned the entire catalogue, mislabelled as one category
The actor sent categoryId, which this gateway ignores: a valid C342 returned all 15,827 products where the storefront shows 919.
The category needs two params sent together — refine=cgid=<id>andtargetCategoryId=<id>. Alone, targetCategoryId is ignored and refine answers 400 CatalogValidationError, which is why neither showed up in isolation. Both now sent; verified live: C342 → 918 products, and the top-level ids match the gateway's own advertised counts (C302 2,615 / C303 2,434 / C301 2,013 / C304 758 / C307 1,107).
Fix: pagination never advanced — every page re-fetched the first
The actor sent start/count; the gateway uses offset/limit, so every request came back at offset: 0 with the default page size. Proven live: offset=100&limit=5 returns offset: 100 with different products, start=100&count=5 returns offset: 0 with 25.
Provenance
Param names and the key=value refine encoding come from the AE app's Retrofit interface rne#b and its nid request model (com.sephora.sephorame, classes3.dex): b(@Query("refine") List<String>, @Query("q"), @Query("sort"), @Query("offset") long, @Query("limit") int, @Query("locale"), @Query("targetCategoryId")).
The cgid refinement in the search response advertises its own valid value set (top-level C301/C302/C303/C304/C307 plus new, bestsellers, gifts-me, …), which is how the pairing was found — sub-category ids like C342 also work once both params are present.
Tests: tests/test_mena_category.py pins the paired category params, the refine encoding and the offset/limit wire. Verified live 2026-08-23 (AE residential proxy): category C342: 918 products, 60 unique records across multiple pages, 0 failed.
Docs
categoryIds was documented as EU-only. It is read by EU, MENA, LATAM, UK and India — each in that market's own id format (SFCC C… ids for EU/MENA/LATAM, numeric ids for UK, category slugs for India). US and SEA ignore it; use category URLs there.
The onlyNewProducts notes gave the wrong reason for MENA's exclusion. A usable mechanism does now exist (the search catalog's new category, ~281 products); it simply is not wired up.
Follow-up available: that new category means MENA can be added to onlyNewProducts — not done here to keep this a pure bug fix. In the meantime it can be crawled directly via categoryIds.
19/08/2026 — v2.6 Incremental refresh: only fetch recently-added products
Feat: onlyNewProducts — skip the catalogue, fetch what's new (19 of 28 markets)
Re-scraping a whole storefront to catch a handful of changes is slow and expensive. Setting onlyNewProducts restricts every category listing to recently-added products: on the US catalogue that is 1,201 products in ~21 listing requests, against 10,784 products / ~180 requests for a full sweep (measured 2026-08-19).
What "new" means depends on what each storefront's API can actually do, and the run logs which of the two applies:
Mechanism
Markets
Result
Filter
US/CA + EU (FR, IT, DE, ES, PL, RO, PT)
Exactly the products Sephora flags as new
Sort
SEA (all 10) + IN
The most recently published, capped by newProductsPageLimit
US/CA send ref=filters[isNew]=true on the v2 catalog endpoint (skincare 2,971 → 383).
EU sends refine_5=c_tagsEnabled=<value> on SFCC product_search. All seven storefronts mark new products with the same internal tag (tag6, established by correlating c_productFlags[].name against each market's localised label), but they do not accept the same refinement value: DE accepts only the raw id tag6enable (its labels return 0), ES/IT/PL/PT only the localised label (the raw id returns 0), FR/RO either. The value is therefore a per-market constant (EU_NEW_TAG) resolved locally, with no discovery round-trip. Verified live on C342 (baseline → filtered): fr 886→113, it 778→101, de 776→91, es 704→107, pl 763→99, pt 704→107, ro 844→120.
SEA and IN cannot filter, only sort, so they use their storefront's own newest-first ordering — ?sort=published_at on SEA's category listing and sort_on=latest on IN's Fynd endpoint. Because a sort returns the whole category merely reordered, both are bounded by the new newProductsPageLimit input (default 3 ≈ 70-110 products per category).
MENA and UK are refused, not silently widened.NEW_PRODUCTS_SUPPORT in src/markets/__init__.py is the single source of truth; markets absent from it are skipped with an ERROR rather than returning their full catalogue at the customer's cost. UK exposes no newness anywhere — filters are price/brand/badge/discount_level (badge = Bundle/Free Gifts/Savings), sorts are rating/price/discount (checked on a 319-result query). MENA advertises a c_tag6enable "New in" refinement (927 of 15,731) and a new-arrival sort id, but refine=, refine_1=, bare params and sort= all fail to apply it.
The filter is carried onto every page of a paginated crawl, and sort markets carry their page cap the same way — dropping either after page 1 would silently widen the run back to the whole category.
Feat: advancedFilter — raw refinement escape hatch (US/CA)
Sends any US refinement expression as ref, e.g. filters[shoppingPreferences]=koreanBeauty. Valid expressions are discoverable in the refinements block of any category response.
A ref= already present on a start URL is honoured too and wins over both inputs; advancedFilter wins over onlyNewProducts. Whenever a lower-precedence source is overridden it is logged, so a filter never vanishes silently.
Both failure modes are silent, and in opposite directions
US with the wrong spelling returns the full set: the expression must be sent as ref=filters[isNew]=true; a bare filters[isNew]=true is accepted and then ignored, with no error. The builder and its tests pin the working form.
EU with a wrong tag value returns total=0, also with no error. The EU search handler therefore warns when a filtered page comes back empty, so a stale tag cannot masquerade as "no new products".
Support was decided per market by applying each mechanism live and watching the count move, never from metadata — a category that does not advertise c_tagsEnabled in its refinements can still be filtered by it (RO C342: unadvertised, still 844→120).
Unchanged wire for existing runs
With no filter set, the US category URL is byte-identical to v2.5 — the frozen v1.x wire is preserved, and a regression test pins it.
Tests: tests/test_only_new_products.py (15 cases) covers the support matrix, the page cap, precedence, request building, the exact upstream URL per market, pagination carry-through, and the frozen unfiltered wire. Sample input at test-new-products.json.
Verified live (2026-08-19, residential proxy): a six-category US incremental run produced 46 records / 46 unique / 0 failed / no URL leaks, and filtered vs unfiltered /shop/skincare return different product sets. EU on C342: eu-fr 8 records and eu-de 13 records (EUR, no URL leaks), exercising both the localised-label and raw-id forms. SEA and IN with newProductsPageLimit=1: sea-nz 21 records (NZD) and in 9 records (INR), both logging the page cap and stopping. MENA and UK URLs produce 0 records and a clear per-market error.
Known issue found during this work, not fixed here
MENA category listing is broken, independently of this feature. The categoryId parameter is ignored entirely — a valid C342 returns 15,731 products (the whole catalogue) where the storefront shows 919 — and _parse_url rejects both live MENA category URL shapes (/shop/{cat}/{sub}/{ID} and the /c/ form). Every MENA category crawl therefore pulls the full catalogue. The correct parameter is still unknown; the AE app's Retrofit interface is the place to look.
18/08/2026 — v2.5.3 US /beauty/ editorial pages supported (bug fix)
Fix: www.sephora.com/beauty/{slug} URLs were rejected as "Invalid / Unsupported url"
Root cause: the US market recognised exactly two start-URL shapes — /shop/{slug} (category) and /product/…-P{id} (detail). /beauty/{slug} matched neither, so the reported URL https://www.sephora.com/beauty/new-skin-care-products?icid2=action_link_new_skincare was dropped before enqueue and the run exited with nothing to scrape.
Why it isn't just a wider regex:/beauty/ pages are not catalog categories. The storefront's own soft links tag them "type": "PageContent" (vs "PageCategory" for /shop/), and the v2 categories endpoint answers their slugs with 404 There is no matching category for :/categories/new-skin-care-products/seo. They are Contentful-backed curated grids served by a separate v1 content endpoint, recovered from the Retrofit interface wu0.b#getBeautyContentPage in com.sephora 26.6 (@GET("content/beauty/{slug}") + @Query("ch") + @Query("loc"), no countryCode) and confirmed live.
New content request kind:src/masker.py gains a content kind, src/auth/us_urls.py builds the real URL with the app's exact param names and order, and a new us-content handler reads data.layout.skuList and enqueues one detail request per product. The other four layout types (buying guide, customer services, directory, open) hold editorial copy, not products — they are skipped with a market-tagged log line rather than failing the run.
Known gap — this emits the page's editorial list, which is probably not what a visitor sees.layout.skuList (9 items for new-skin-care-products) is an SSR/editorial list. The page's visible grid is a separate Constructor.io-ranked feed at https://www.sephora.com/gway/v1/dotcom-sys/browse/{layout.ctorPodId} — same collection (display_name "New Skincare", pod id supplied by the page itself, and the only product call the page makes) but a substantially different set: only 1 of the 9 pinned ids appears in the feed's first 100, and the feed is broader (lip balms, value sets, a hand sanitizer). It is not "the rest of the page" — the two lists barely intersect. Verified 2026-08-19 by rendering the page: the visible grid is the feed (first tile P480529 Touchland Power Mist Hand Sanitizer, then Supergoop! PLAY, Vacation Mini Classic Whip, rhode Glazing Milk, rhode Highlight Milk — the feed's order, none of them in skuList), so what this actor emits for a /beauty/ URL is 9 products a visitor does not see there.
Why it is not wired up yet (investigated 2026-08-18). It is reachable — earlier notes here blamed Kasada, then Akamai; both were wrong, and the TCP drops were self-inflicted by probing with the app User-Agent over a Chrome JA3. The feed returns HTTP 200 to plain curl_cffi with the market's ordinary Seph-Access-Token. What stops it being a drop-in win: it caps at 200 results per sweep at any page size and reorders between requests (~160-180 unique of 383; 216 across two sweeps), and it returns nothing for the bestsellers pods on these parameters. So switching to it is a product decision — a different, larger, non-deterministic set — not a bug fix. For a guaranteed-complete assortment, use the matching /shop/… category URL, which paginates fully.
Canada:sephora.com/ca/en/beauty/… and /ca/fr/… resolve through the same path and produce CAD records with /ca/en canonical URLs.
Output hygiene:source.crawlUrl on every record stays the public /beauty/ page the customer passed (tracking params intact); the v1 content endpoint never reaches the dataset or the logs.
Scope note: only /beauty/{slug} is claimed. A multi-segment /beauty/{a}/{b} (e.g. /beauty/guides/how-to-apply-blush) is still reported unsupported — the app declares the slug as an @Path, which percent-encodes slashes, so that is not a shape the upstream API can serve.
Tests: URL classification (incl. the reported URL verbatim, both CA locales, trailing slash, multi-segment rejection), /shop/ and PDP regression cases, synthetic-request routing, the exact upstream URL, and handler behaviour across grid / non-grid / empty / 404 pages are locked in (tests/test_us_content_pages.py, 15 cases). The e2e manifest schema gained a content URL shape, wired so the operator runner picks it up automatically.
Verified live (2026-08-18, US residential proxy): the reported URL produced 9 records, 0 failed requests, every crawlUrl the public /beauty/ page and every canonicalUrl a public /product/ page. CA (/ca/en/beauty/…, CA proxy): 3 records, CAD, 0 failed.
11/08/2026 — v2.5.2 UK slug PDP URLs resolved (bug fix)
Fix: www.sephora.co.uk/p/{slug} product URLs failed with G-2 "could not be found" (HTTP 400)
Resolution hop: a /p/{slug} URL is now detected by token shape (P\d{4,} = id, else slug) and routed through a new resolve request that queries /catalog/v1/products?query={slug-as-text} (the only search param the gateway honors — q/term/keyword return a static suggestion set) and matches the slug against each result's normalized {brand} {name} (exact / prefix / ≥0.6 token overlap). The winning P… id is enqueued as a normal detail request. No confident match → recorded not-found, never a wrong product.
Output fix (source.canonicalUrl): the record now carries the real public /p/{slug} page the request started from, not a /p/{productId} deeplink or the internal fetch endpoint.
Proxy: UK reads only clear Akamai from a GB residential IP (datacenter and reputation-flagged residential exits get 403). A market may now declare preferred_proxy_country; when every active market agrees and the customer left the country unset, the run defaults to it (UK → GB). Mixed runs and an explicit proxy.apifyProxyCountry are untouched.
Latency: the Akamai-blocked /login/v1/login burst is no longer re-run on every catalog request — after one failed attempt it backs off for 5 min (reads don't need the session), saving ~6s + three dead round-trips per call.
Tests: slug-vs-id parsing, slug→id matching (incl. the reported URL), request routing, canonicalUrl threading, and the proxy-country pin are locked in (tests/test_uk_resolution.py, 21 cases).
Verified live (2026-08-11, residential proxy, country auto-pinned to GB): the reported URL https://www.sephora.co.uk/p/rhode-pocket-blush-buildable-hydrating-cream-blush resolved to P1000209341 and produced a valid record (title, 12 variants, 5 images, GBP, canonical = the public slug URL) — 2 requests, 0 failed.
07/08/2026 — v2.5.1 MENA storefront URLs accepted (bug fix)
Fix: www.sephora.me product URLs were rejected as "Invalid / Unsupported url"
Root cause: the six MENA markets were registered under per-country hostnames (www.sephora.ae, www.sephora.sa, …) inferred during Phase 1 recon and never verified live. The real consumer storefront is the single host www.sephora.me with a /{cc}-{lang}/ path prefix — the AE app's own declared production web base. Unregistered hostnames fall back to the us market, which then rejected every MENA URL and tagged the error [us].
Routing:detect_market now resolves www.sephora.me (and the apex) by path prefix — /ae-en/, /sa-en/, … → mena-ae, mena-sa, …. Bare (/ae/) and Arabic (/ae-ar/) prefixes resolve to the same market. An absent or unknown prefix defaults to AE rather than falling through to us. The legacy per-country hostnames stay registered.
URL parsing: the live grammar joins the product id with a slash (/p/{slug}/P{id}); the parser only accepted the hyphen form ({slug}-P{id}). Both now parse. ?productVariantId= is ignored — every variant is already emitted.
Output fix (source.canonicalUrl): MENA records carried a wrong host and a doubled slash (https://www.sephora.ae/ae//mega-mix-palette/P10061352). They now emit the real page URL, e.g. https://www.sephora.me/ae-en/p/mega-mix-palette/P10061352. This changes existing MENA output — consumers keying on the old value must update.
Diagnosability: an unrecognized hostname now logs a warning naming the host and the fallback market, once per host, instead of failing silently under the wrong market's tag.
All families: apex hostnames now resolve both directions. Previously only www.x → x was tried, so e.g. https://sephora.fr/... fell through to us; it now resolves to eu-fr. Same bug class as the MENA fault above.
Tests: the five reported URLs are locked in as routing + parse regressions; all six MENA path prefixes are covered; the canonicalUrl assertion is now exact rather than a prefix check (the prefix check is what let the malformed URL ship). README market table and the AE e2e manifest updated to the real grammar.
Verified live (2026-08-07, residential proxy, country-pinned): all 6 MENA markets PASS — 28 records, 0 failed requests, every canonicalUrl on the real host with the correct locale prefix. Run: tests/e2e/runs/mena-20260807T141315Z.jsonl.
Fix: live e2e gate was red for every market since v2.5 (unrelated pre-existing bug)
tests/e2e/scripts/assert_dataset.py kept its own hardcoded copy of the record's top-level key list. v2.5 added size to dataset_schema.json, all seven market transforms, and test_dataset_schema_compat.py — but not to this copy, so the script rejected every record with unexpected keys: ['size']. Any Layer-3 run since 30/06/2026 would have failed on all markets, not just MENA.
The script now derives the key set from .actor/dataset_schema.json instead of duplicating it, so the schema stays the single source of truth and this drift cannot recur.
NEWoptions[].values[].description — per-shade undertone/finish text, mapped from the US API's variationDesc (e.g. "light, neutral peach", "with neutral undertones", "medium warm 4"). Colour shades only; blank for size/scent variations.
NEW top-level size — headline size/volume of the default SKU (US: currentSku.size, e.g. "1 oz / 30 mL"). Non-US markets emit "" (their SFCC/GraphQL APIs don't expose a clean product-level size).
NEWvariants[].size — per-SKU size/volume (US-only; ancillary minis differ from the headline size).
Scope: undertone/finish + size are US/CA only — the international storefronts (SFCC EU/LATAM/MENA, GraphQL UK/SEA/IN) carry no per-shade description, so description is absent and size is "" there.
Schema docs updated: dataset_schema.json, test_dataset_schema_compat.py, and API_DOCUMENTATION.md.
09/05/2026 — v2.4 Phase 4 IN ship + Phase 3 UK reconciliation
Feat: Phase 4 — IN (Sephora India on Fynd Platform)
Added in family — single market on api.sephora.cn's Reliance-owned cousin api.fynd.com. Static Bearer auth (Base64 of compile-time applicationId:applicationToken recovered from com.jio.sephora 1.0.34). Path grammar /service/application/catalog/v1.0/products/{slug}/. Slug-based product addressing, page_no/page_size pagination. Module name in_market (avoids Python in keyword); family slug "in" in _FAMILY map.
Added latam-mx (es-MX, MXN) and latam-br (pt-BR, BRL) on apps.sephora.eu with /latam/ URL prefix and OCAPI v23_2 (newer than EU's v18_3). Single SODA key + DW client ID across both countries; per-country UA via LATAM_VERSIONS_BY_COUNTRY. Recovered from latam.sephora.sephoramx 26.4.3 + latam.sephora.sephorabr 26.4.2.
Feat: Phase 3 — UK (initial scaffold)
Added uk family — single market on apps.sephora.eu with a flatter REST surface (/catalog/v1/, /login/v1/, /content/v1/) than EU's OCAPI commerceCloud paths. Recovered from co.uk.sephora 19.7.1 (React Native + Hermes + Akamai BMP). x-consumer-id: AOC-ANDROID-UK (vs EU's AOC-ANDROID-EME).
08/05/2026 — v2.2 Phase 1 MENA ship
Feat: Phase 1 — MENA (6 countries on SCAPI/Apigee)
Added mena-ae, mena-sa, mena-bh, mena-om, mena-kw, mena-qa on apps.sephora.me (Apigee gateway + SCAPI). Per-country routing via lowercase apikey header carrying per-country Apigee key (AE + SA have their own; BH/OM/KW/QA fall back to AE's, mirroring how the App Store routes them). Six countries served by one backend. Recovered from ae.sephora.sephorauae 3.45.0-5386 + sa.sephora.sephorasa 3.45.0-5385.
18/04/2026 — v2.1 Crawlee migration + URL masking
Unified Crawlee HttpCrawler across US + EU + SEA with synthetic-URL masking, global circuit breaker, per-market pre-flight, declarative retry policy, serialized EU guest-token refresh, EU product-ID case normalization, maxProducts cap honored in EU pagination; CZ, GR, and PPE instrumentation removed, PT now served via the Spain backend with pt-PT locale.
NEWvariants[].wishlisted boolean field (SEA-only; analog of US lovesCount).
Feat:
Multi-market support — US + Canada + 9 EU countries + 10 APAC countries in one actor.
Auto-detect market from startUrls hostname; no input changes required for existing US users.
Optional market and locale override inputs for edge cases (bare IDs, region mismatches).
Per-market observability counters persisted to the default KV store as run-summary.
Architecture:
Refactored src/main.py into a lightweight dispatcher. Per-region logic lives in src/markets/{us,eu,sea}.py.
EU and SEA ports use curl_cffi for upstream compatibility. US keeps Crawlee's HttpCrawler with the existing session-pool auth flow unchanged.
Strict market-module isolation — per-region auth state cannot cross-contaminate.
Compatibility:
Existing US run configs work unchanged. Pre-v2.0 output fields are preserved; only the additive market key is new.
Standalone autofacts/sephora-eu-scraper and autofacts/sephora-nz-scraper listings remain live but are pinned to their current version. New bug fixes and features land in this repo only.
Deps:
Added: curl_cffi[async] >= 0.7, lxml >= 5.0.
10/03/2026
Feat: add sentiments field to product output, merging sentiment summary and sentiment items
