Talabat Mart Scraper (GCC q-commerce grocery) — MVP: Dubai

First maintained, structured, PPE, agent-ready Talabat Mart dark-store grocery SKU + price Actor. Play ③ uncovered-in-demand (existing Apify talabat-scraper actors are RESTAURANT menus, not q-commerce grocery). Single retailer + single city to start; schema/architecture reserve multi-retailer (Noon Minutes / Careem Quik) + multi-city.

Pipeline (3 stages, all pure HTTP — confirmed by recon 2026-07-23)

1. AREA GET /uae/groceries/<areaId>/<slug> → __NEXT_DATA__.props.pageProps.talabatMartData.branchId = the Talabat Mart dark store for that area. Dedup branchIds across areas → the Dubai tMart set.
2. STORE GET /uae/grocery/<branchId>/<slug>?aid=<areaId> → the category-listing links. (The ?aid=<areaId> param is required — without it the store URL 307-redirects to /uae.)
3. CATALOG GET /uae/grocery/<branchId>/<slug>/<category>/<subcategory>?aid=<areaId> (Next.js grocery-items page) → the full item list is SSR'd into __NEXT_DATA__.props.pageProps.initialState. No BFF, no browser.

Confirmed item schema (per item in the SSR payload):

{ id (uuid), sku, title, slug, description, originalPrice, price, discount, discountPercentage, image, images[], stockAmount, attributes[] }

Anti-bot (determined by recon 2026-07-23, and VALIDATED live)

• Edge = Cloudflare Bot Management — permissive; static bot-UA blocklist; no JS challenge (a real browser from a clean IP passed instantly; a python-requests UA → 403).
• Session = Delivery Hero Perseus cookies + __cf_bm + tlb_country; catalog geo-gated to UAE.
• ⇒ HTTP crawler (got-scraping browser-TLS, no headless browser) + realistic mobile UA
  • RESIDENTIAL-AE proxy + SessionPool + moderate rate. See src/crawler.ts.
• Note: catalog + prices render even from a non-AE IP, but accurate stockAmount needs a UAE IP → residential-AE is required, not optional.

Run

npm install
npm run build
apify run -p --input-file test.json      # default = 1 zone (Al Badaa) × 50 products, fast + cheap

Default input is intentionally small/fast — the Apify daily auto-tester runs it; a slow/failing default flags the Actor UNDER_MAINTENANCE.

Monetization (Console-only config)

Pure PPE + limited permissions + structured output → auto agentic=true (x402/MCP funnel).

• main event: per-product-result ~$2–4 / 1k SKU
• aux event: per-store/zone start ~$0.02
• free top-N SKU sample (wedge, mirrors Actowiz's free PoC)

Observability (Scraper Insight)

Every fetch is reported to the Scraper Insight hub (https://insight.auto7lab.com, override via INSIGHT_URL) with machine auth (X-App-ID + X-App-Secret, this Actor's registered app; override via INSIGHT_APP_ID / INSIGHT_APP_SECRET): successes with real status + latency from src/crawler.ts's requestHandler, exhausted-retry failures from failedRequestHandler with an HTTP-style classified status (src/insight.ts — Cloudflare block throws carry (status 403) and classify accordingly, timeouts → 504). Reporting is fire-and-forget: a dead Insight endpoint never fails the crawl. This feeds the dashboard, alert rules, and the Cloudflare-escalation health signal.

Validation gate (do NOT skip)

Off-platform demand (Actowiz dashboards, LinkedIn basket use-case) proves interest, not that buyers convert to a self-serve Actor. ≥ 8–10 distinct paying accounts within 60 days of listing, or downgrade / re-scope. Day-20 leading indicator: free-sample pulls ≈ 0 → fail fast.

Remaining before scale

• Decode the useful attributes[] characteristics beyond barcode (weight/volume/returnable) if needed.

Area list maintenance

src/dubaiZones.ts (153 Dubai areas, from Talabat's UAE areas sitemap) is the single source of truth. The Console multiselect for areaIds is generated from it — after refreshing the zone map from the sitemap, re-sync the input schema:

$npm run build && node scripts/sync-area-enum.mjs

categories stays free-form on purpose: category slugs are per-store and discovered at runtime, so there is no closed vocabulary to enumerate.

Icon

.actor/icon.png (256×256) is Talabat's own favicon (brand orange #f55905), the mark that identifies what this Actor scrapes. Apify has no actor.json icon field — set it in Console → Publication → Display information → Icon by uploading this file.

Output — shared spec Product (src/spec.ts)

Records follow the fleet-wide product spec (same contract as farfetch / asos / boohoo / ulta): source / title / description / details / brand / categories / medias / variants / price / extraInfo. Fleet conventions:

• Prices in minor units (fils, AED × 100): price.current = 1250 = AED 12.50; price.currentFormatted = "AED 12.50"; previous falls back to current when there is no was-price; a real was-price also fills original.
• One record per product-offer at one dark store. source.id = the Talabat global-catalog uuid (cross-store product identity), so the same product at two dark stores = two records; offer uniqueness = (source.id, extraInfo.darkStore.id).
• Grocery = single-variant products: variants has exactly one entry carrying the retailer sku and the same price.
• extraInfo carries the q-commerce dimensions the spec has no first-class slot for: scrapedAt, city, darkStore {id, name, zone, areaId}, barcode (EAN/UPC — the cross-retailer matching key), size, discountPercentage, superSaver,

  fulfilment {deliveryFeeAed, etaMinutes, minOrderAed}
  .

Output hygiene

Everything fetched is a public SSR page, so URLs are already public (no hidden BFF/token to leak). source.canonicalUrl = the public product detail page /uae/grocery/<branchId>/<storeSlug>/product/<slug>/s/<sku>?aid=<areaId> (confirmed live 2026-07-30; falls back to the category page if an item ever lacks slug/sku). source.crawlUrl = the public category page the item was parsed from. The ?aid=<areaId> query param is part of the public URL scheme (pages 307-redirect without it), not an internal token. Keep it that way.