Google Shopping Immersive
Pricing
from $1.00 / 1,000 results
Scrapes basic product details from a Google Shopping immersive product URL containing prds.
Pricing
from $1.00 / 1,000 results
Rating
0.0
(0)
Developer
Abinash Sonowal
Maintained by CommunityActor stats
1
Bookmarked
3
Total users
2
Monthly active users
6 days ago
Last modified
Categories
Share
# apify push starts from .gitignore; re-include the build output it ignores,# and skip uploading source that the image doesn't use (Docker runs dist/ only).src/!dist/node_modules/dist/
.env.env.*!.env.example
.idea/.vscode/
.DS_StoreThumbs.db
apify_storage/storage/crawlee_storage/1# AGENTS.md — AI Agent Onboarding2
3> Read this file first. It gives you everything you need to understand and work on this repo without reading every file.4
5---6
7## What this repo is8
9**Apify actors** that scrape Google Shopping. The repo has **three independent branches**, each deployed as a separate actor:10
11| Branch | Actor purpose | Input | Output |12|---|---|---|---|13| `main` | Base / shared utilities | — | — |14| `shopping` | Search-results scraper | Search query + country | List of ~40 product listings |15| `immersive` | Product detail scraper | Product URL (with `prds=`) | Single product: specs, sellers, competitors |1617You are currently on the **`immersive`** branch unless told otherwise. Do not mix logic from other branches.18
19---20
21## Architecture (`immersive` branch)22
23```24src/25 main.js # Entry: Actor lifecycle, input validation, URL building, fetch + block detection26 parser.js # Pure HTML parsing (cheerio) — no network, no Apify calls27 proxy_http_client.js # The ONLY HTTP client: got-scraping + Apify residential proxy, with retries28.actor/29 actor.json # Actor metadata30 input_schema.json # Defines: url (required), country (default "in")31 dataset_schema.json # Output schema32Dockerfile # apify/actor-node:20, installs package.json33package.json # apify, cheerio, got-scraping (ESM, "type": "module")34```35
36### Data flow37```38Actor.getInput()39 → url, country40 → validateInput(url, country)41 → Actor.createProxyConfiguration({ groups: ["RESIDENTIAL"], countryCode: country.toUpperCase() })42 → new ProxyHttpClient(proxyUrl)43 → buildImmersiveUrl(url, country) # injects ibp, hl, gl, udm params44 → httpClient.fetch(url) # got-scraping GET via proxy → [html, finalUrl]45 → isBlocked(html)? → log warning, return null46 → parseProduct(html, url, finalUrl) # cheerio47 → Actor.pushData(product)48```49
50This is a JavaScript (Node.js, ESM) Apify actor. Run locally with `apify run` or `npm start`.51
52---53
54## Key classes & functions55
56### `src/main.js`57- **`validateInput(url, country)`** → throws on missing/invalid url, non-https, non-Google domain, missing `prds`, or bad country code58- **`buildImmersiveUrl(url, country)`** → adds `ibp=oshop`, `hl=en`, `gl=<country>`, `udm=28` to the input URL (only if absent)59- **`runImmersive(httpClient, url, country)`** → orchestrates fetch + parse, saves `sparse-response.html` if result is empty60- top-level `Actor.init()` / `Actor.exit()` lifecycle → Apify actor entrypoint61
62### `src/parser.js` — exported functions63- `parseProduct(html, url, finalUrl)` → returns the full output object (loads cheerio internally)64- `isBlocked(html)` → checks for CAPTCHA / "unusual traffic" signals65- internal helpers: title (`data-attrid="product_title"` → `<title>`), specs (`data-attrid="product_attributes_facet"`), sellers (`data-merchant-name`), current price (`data-crcy` + `aria-label` "Current price:"), old price (`aria-label` "Old price was"/"Maximum retail price:"), filters (`data-pvf` across visible + injected DOM)66
67### `src/proxy_http_client.js`68- **`ProxyHttpClient`** — the only HTTP client; constructed with a `proxyUrl`; sends full browser-like headers via `got-scraping`; `fetch(url)` returns `[html, finalUrl]` and retries up to 3× with exponential backoff69
70---71
72## Output schema (single item pushed to dataset)73
74```ts75{76 "input_url": string, // URL from actor input77 "final_url": string, // Normalized Google immersive URL fetched78 "title": string | null,79 "description": string | null,80 "images": string[], // Product image URLs, primary image first81 "rating": number | null, // e.g. 4.682 "review_count": number | null, // e.g. 11000 (handles "11k" → 11000)83 "features": Record<string, string>, // e.g. {"Storage": "256 GB", "Color": "Black"}84 "filters": [{85 "category": string, // e.g. "Colour", "Capacity"86 "options": [{87 "name": string,88 "selected": boolean,89 "image": string | null, // swatch image URL for colour variants (omitted if absent)90 }],91 }],92 "buying_options": [{93 "merchant": string | null,94 "merchant_id": string | null,95 "offer_id": string | null,96 "title": string | null,97 "price": string | null, // e.g. "₹79,900"98 "currency": string | null, // ISO 4217 e.g. "INR"99 "old_price": string | null,100 "target_url": string | null,101 "status": string | null, // e.g. "In stock"102 "delivery": string | null,103 "offer_rating": number | null,104 "seller_logo": string | null,105 }],106}107```108
109---110
111## Input schema112
113```ts114{115 "url": string, // Required. Google Shopping URL with prds= param116 "country": string, // Optional. Default "in". 2-letter ISO e.g. "us", "gb"117}118```119
120---121
122## Gotchas & conventions123
1241. **`prds` param is required** — `validateInput()` throws if missing from the input URL1252. **Block detection** — checks for `"before you continue"`, `"unusual traffic"`, `"verify you're human"`, `"g-recaptcha"` in page text1263. **Sparse response** — if parsed product has no `features` AND no `buying_options`, the raw HTML is saved to the KV store as `sparse-response.html` for debugging1274. **Currency extraction** — `extractCurrency()` first checks symbol map (`₹→INR`, `$→USD`, `€→EUR`, `£→GBP`, `¥→JPY`), then regex for 3-letter uppercase ISO code1285. **Review count parsing** — handles `"11k"` → `11000` and `"1.2m"` → `1200000`1296. **Single HTTP client** — there is exactly one client, `ProxyHttpClient` (`src/proxy_http_client.js`); there is no abstract base or local/no-proxy client1307. **No Playwright / browser** — this actor is HTTP-only (got-scraping); do not add Playwright here1318. **Proxy group** — must use `RESIDENTIAL` group; `DATACENTER` group typically gets blocked by Google1329. **Retry logic** — `ProxyHttpClient.fetch()` retries up to 3 times with exponential backoff (1s, 2s, 4s) on any request error; handles transient `UPSTREAM502` / `590` errors from the residential proxy tier133
134---135
136## Running locally137
138```bash139# Install deps140npm install141
142# Run the actor locally with Apify input143apify run144
145# Run directly146npm start147```148
149---150
151## Branch-specific reminders152
153- **`immersive`** (this branch): input = product URL, output = 1 item with nested sellers/specs154- **`shopping`** branch: input = search query `q`, output = list of ~40 flat product listings155- Do **not** merge logic between branches — they are independent deployments# Apify Node.js base image.FROM apify/actor-node:20
# Install ONLY the runtime dependencies, from the build-generated dist/package.json# (which contains no `workspace:` specs — npm rejects those).COPY dist/package.json ./package.jsonRUN npm --quiet set progress=false \ && npm install --omit=dev --omit=optional \ && echo "Installed NPM packages:" \ && (npm list --omit=dev --all || true) \ && echo "Node.js version:" \ && node --version
# Copy the self-contained bundle (src/ + @gs/shared inlined by esbuild).COPY dist ./dist
CMD ["node", "dist/main.js"]{ "name": "@gs/immersive", "version": "0.0.1", "type": "module", "description": "Scrapes product details from a Google Shopping immersive URL. Also exports parseProduct for reuse.", "engines": { "node": ">=18" }, "main": "src/immersive.js", "exports": { ".": "./src/immersive.js" }, "scripts": { "start": "node dist/main.js", "dev": "node src/main.js" }, "dependencies": { "apify": "^3.2.6", "cheerio": "^1.0.0", "got-scraping": "^4.0.6" }, "devDependencies": { "@gs/client": "workspace:*" }}{ "$schema": "https://apify.com/schemas/v1/actor.ide.json", "actorSpecification": 1, "name": "google-shopping-immersive", "title": "Google Shopping Immersive", "description": "Scrapes basic product details from a Google Shopping immersive product URL containing prds.", "version": "0.0", "buildTag": "latest", "input": "./input_schema.json", "dockerfile": "../Dockerfile"}{ "actorSpecification": 1, "fields": { "input_url": { "title": "Input URL", "type": "string", "description": "Original input URL" }, "final_url": { "title": "Fetch URL", "type": "string", "description": "Normalized Google immersive URL fetched by the actor" }, "title": { "title": "Title", "type": "string", "description": "Product name" }, "description": { "title": "Description", "type": "string", "description": "Product description" }, "images": { "title": "Images", "type": "array", "description": "Product image URLs, primary image first" }, "rating": { "title": "Rating", "type": "number", "description": "Average rating out of 5" }, "review_count": { "title": "Review count", "type": "integer", "description": "Total number of reviews" }, "features": { "title": "Features", "type": "object", "description": "Product attributes/specifications" }, "filters": { "title": "Filters", "type": "array", "description": "Variant filters (e.g. Colour, Capacity) with available options, selected state, and a per-option immersive URL (the option's pvf token swapped into prds)" }, "buying_options": { "title": "Buying options", "type": "array", "description": "Merchant offers with price, currency, delivery info, seller logo, and offer title" } }}{ "$schema": "https://apify.com/schemas/v1/input.ide.json", "title": "Google Shopping Immersive Scraper", "type": "object", "schemaVersion": 1, "properties": { "url": { "title": "Product URL", "type": "string", "description": "Google Shopping immersive product URL containing a prds query parameter", "editor": "textfield", "default": "https://www.google.com/search?ibp=oshop&q=Apple+iPhone+16+(128GB+Storage,+Ultramarine)&prds=catalogid:3996416311311992647,gpcid:1465601687215425854,headlineOfferDocid:12871151456716918231,imageDocid:9363746595791942731,mid:576462796102771346,productid:16690412950309539714,pvo:25,pvt:hg,rds:PC_1465601687215425854%7CPROD_PC_1465601687215425854&pvorigin=25&hl=en&gl=in" }, "country": { "title": "Country Code", "type": "string", "description": "Google country code for localized results (e.g. us, gb, in)", "default": "in", "editor": "textfield" } }, "required": []}{ "openapi": "3.0.0", "info": { "title": "Google Shopping Immersive Scraper", "version": "1.0.0", "description": "Scrapes details of a specific product from Google Shopping immersive page using a product URL." }, "paths": { "/run-sync-get-dataset-items": { "post": { "operationId": "runScraper", "summary": "Run scraper and return product details", "requestBody": { "required": true, "content": { "application/json": { "schema": { "type": "object", "required": ["url"], "properties": { "url": { "type": "string", "description": "Google Shopping immersive product URL" }, "country": { "type": "string", "description": "2-letter country code e.g. in, us, gb", "default": "in" } } } } }, "responses": { "200": { "description": "Scraped product details", "content": { "application/json": { "schema": { "type": "object", "properties": { "input_url": { "type": "string" }, "final_url": { "type": "string" }, "title": { "type": "string", "nullable": true }, "description": { "type": "string", "nullable": true }, "images": { "type": "array", "items": { "type": "string" } }, "rating": { "type": "number", "nullable": true }, "review_count": { "type": "integer", "nullable": true }, "features": { "type": "object", "nullable": true }, "filters": { "type": "array", "items": { "type": "object" }, "nullable": true }, "buying_options": { "type": "array", "items": { "type": "object" }, "nullable": true }, "competing_products": { "type": "array", "items": { "type": "object" }, "nullable": true } } } } } } } } } } }}{ "actorSpecification": 1, "views": { "overview": { "title": "Immersive product", "transformation": { "fields": ["title", "rating", "review_count", "features", "buying_options", "final_url"] }, "display": { "component": "table", "properties": { "title": { "label": "Product" }, "rating": { "label": "Rating" }, "review_count": { "label": "Reviews" }, "features": { "label": "Features" }, "buying_options": { "label": "Buying options" }, "final_url": { "label": "Fetch URL", "format": "link" } } } } }}1// Bundled by esbuild (src/ + @gs/shared inlined). Do not edit — run `pnpm build`.2
3// immersive/src/main.js4import { Actor, log as log2 } from "apify";5
6// client/proxy_http_client.js7import { log } from "apify";8import { gotScraping } from "got-scraping";9var MAX_RETRIES = 3;10var RETRY_BACKOFF_BASE_MS = 1e3;11var DEFAULT_HEADERS = {12 "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",13 Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",14 "Accept-Language": "en-IN,en;q=0.9",15 "Cache-Control": "no-cache",16 Pragma: "no-cache",17 Priority: "u=0, i",18 "Sec-Ch-Ua": '"Google Chrome";v="137", "Chromium";v="137", "Not/A)Brand";v="24"',19 "Sec-Ch-Ua-Mobile": "?0",20 "Sec-Ch-Ua-Platform": '"Windows"',21 "Sec-Fetch-Dest": "document",22 "Sec-Fetch-Mode": "navigate",23 "Sec-Fetch-Site": "none",24 "Sec-Fetch-User": "?1",25 "Upgrade-Insecure-Requests": "1"26};27var sleep = (ms) => new Promise((resolve) => {28 setTimeout(resolve, ms);29});30var ProxyHttpClient = class {31 constructor(proxyUrl, headers = null) {32 this.proxyUrl = proxyUrl;33 this.headers = headers || DEFAULT_HEADERS;34 }35 /**36 * @param {string} url37 * @returns {Promise<[string, string]>} `[htmlBody, finalUrl]`38 */39 async fetch(url) {40 let lastErr;41 for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {42 try {43 const response = await gotScraping({44 url,45 proxyUrl: this.proxyUrl,46 headers: this.headers,47 useHeaderGenerator: false,48 https: { rejectUnauthorized: false },49 timeout: { request: 3e4 },50 retry: { limit: 0 }51 });52 return [response.body, response.url];53 } catch (err) {54 lastErr = err;55 if (attempt < MAX_RETRIES) {56 const delay = RETRY_BACKOFF_BASE_MS * 2 ** (attempt - 1);57 log.warning(58 `Proxy/HTTP error on attempt ${attempt}/${MAX_RETRIES} (${err.message}). Retrying in ${delay / 1e3}s...`59 );60 await sleep(delay);61 } else {62 log.error(`All ${MAX_RETRIES} attempts failed. Last error: ${err.message}`);63 }64 }65 }66 throw lastErr;67 }68};69
70// client/is_blocked.js71import * as cheerio from "cheerio";72function isBlocked(html) {73 if (html.toLowerCase().includes("g-recaptcha")) return true;74 const text = cheerio.load(html).text().toLowerCase();75 return [76 "before you continue",77 "unusual traffic",78 "verify you're human"79 ].some((s) => text.includes(s));80}81
82// immersive/src/immersive.js83import * as cheerio2 from "cheerio";84
85// immersive/src/selectors.js86var SELECTORS = {87 /** Product title container. */88 title: '[data-attrid="product_title"]',89 /** Product description container. */90 description: '[data-attrid="product_description"]',91 /** Inner text element of the description, when present. */92 descriptionText: "#description_container",93 /** One spec row per element, text formatted as "key: value". */94 spec: '[data-attrid="product_attributes_facet"]',95 /** Variant filter options (colour, storage, ...). */96 filterOption: "[data-pvf]",97 /** Filter list container whose aria-label names the category ("Color options"). */98 filterList: '[role="list"][aria-label]',99 /** Fallback category heading preceding a filter option. */100 filterHeading: '[role="heading"]',101 /** Gallery images with a lazy-loaded data-src. */102 galleryImage: "[data-item-index][data-src]",103 /** Main product image meta tag. */104 mainImageMeta: 'meta[property="og:image"]',105 /** Offers grid holding all seller cards. */106 offersGrid: '[data-attrid="organic_offers_grid"]',107 /** New-layout offer cards inside the offers grid. */108 offerCardNew: '[jsname="uwagwf"][role="listitem"]',109 /** Old-layout merchant elements carrying data-merchant-name / data-target-url. */110 merchantOld: "[data-merchant-name]",111 /** Old-layout offer card ancestor of a merchant element. */112 offerCardOld: '[role="listitem"]',113 /** New-layout merchant name: attribute carrier and text fallback. */114 merchantContext: "[data-report-feedback-about-context]",115 merchantName: ".gUf0b",116 /** Offer title within a seller card. */117 offerTitle: ".rYkzq.y1FcZd",118 /** Offer stock/status line ("In stock"). */119 offerStatus: ".OaQPmf",120 /** New-layout seller rating badge. */121 offerRatingNew: ".NFq8Ad",122 /** Price container carrying the data-crcy currency attribute. */123 priceContainer: "[data-crcy]",124 /** Outbound merchant link in a seller card. */125 externalLink: 'a[href^="http"]'126};127
128// immersive/src/pvf.js129var FIELD_SELECTION = 18;130var FIELD_CATEGORY = 34;131var FIELD_VALUE = 42;132function varint(n) {133 const bytes = [];134 while (n > 127) {135 bytes.push(n & 127 | 128);136 n >>>= 7;137 }138 bytes.push(n);139 return bytes;140}141function lenDelimited(tag, payload) {142 return [tag, ...varint(payload.length), ...payload];143}144function encodePvf(selections) {145 const out = [];146 for (const { category, value } of selections) {147 const cat = [...Buffer.from(category.toLowerCase(), "utf8")];148 const val = [...Buffer.from(value.toLowerCase(), "utf8")];149 out.push(...lenDelimited(FIELD_SELECTION, [150 ...lenDelimited(FIELD_CATEGORY, cat),151 ...lenDelimited(FIELD_VALUE, val)152 ]));153 }154 return Buffer.from(out).toString("base64url");155}156function readVarint(buf, i) {157 let n = 0;158 let shift = 0;159 while (i < buf.length) {160 const b = buf[i++];161 n |= (b & 127) << shift;162 if ((b & 128) === 0) return [n, i];163 shift += 7;164 }165 return [n, i];166}167function decodePvf(token) {168 if (!token) return [];169 let buf;170 try {171 buf = Buffer.from(token, "base64url");172 } catch {173 return [];174 }175 const selections = [];176 let i = 0;177 while (i < buf.length) {178 const tag = buf[i++];179 if ((tag & 7) !== 2) return [];180 let len;181 [len, i] = readVarint(buf, i);182 const end = i + len;183 if (end > buf.length) return [];184 if (tag === FIELD_SELECTION) {185 const sel = { category: "", value: "" };186 while (i < end) {187 const t = buf[i++];188 let l;189 [l, i] = readVarint(buf, i);190 const s = buf.toString("utf8", i, Math.min(i + l, end));191 if (t === FIELD_CATEGORY) sel.category = s;192 else if (t === FIELD_VALUE) sel.value = s;193 i += l;194 }195 selections.push(sel);196 }197 i = end;198 }199 return selections;200}201function withSelection(selections, category, value) {202 const cat = category.toLowerCase();203 const val = value.toLowerCase();204 const next = selections.map((s) => s.category === cat ? { category: cat, value: val } : s);205 if (!next.some((s) => s.category === cat)) next.push({ category: cat, value: val });206 return next;207}208function pvfFromUrl(url) {209 if (!url) return null;210 let prds;211 try {212 prds = new URL(url).searchParams.get("prds");213 } catch {214 return null;215 }216 if (!prds) return null;217 const part = prds.split(",").find((p) => p.startsWith("pvf:"));218 return part ? part.slice("pvf:".length) : null;219}220
221// immersive/src/immersive.js222var CURRENCY_BY_SYMBOL = {223 "\u20B9": "INR",224 "$": "USD",225 "\u20AC": "EUR",226 "\xA3": "GBP",227 "\xA5": "JPY"228};229function cleanText(value) {230 if (!value) return null;231 return value.replace(/\s+/g, " ").trim();232}233function nodeText(el, sep = "") {234 if (!el) return "";235 const parts = [];236 const walk = (node) => {237 for (const child of node.children || []) {238 if (child.type === "text") {239 const t = child.data.trim();240 if (t) parts.push(t);241 } else if (child.type === "tag") {242 walk(child);243 }244 }245 };246 walk(el);247 return parts.join(sep);248}249function findPrevious($, el, selector) {250 const all = $("*").toArray();251 const idx = all.indexOf(el);252 if (idx < 0) return null;253 for (let i = idx - 1; i >= 0; i--) {254 if ($(all[i]).is(selector)) return all[i];255 }256 return null;257}258function findByAria($, $scope, pred) {259 return $scope.find("[aria-label]").filter((_, e) => pred($(e).attr("aria-label") || "")).first();260}261function extractInjectedSoup($) {262 let injectedHtml = "";263 $("script").each((_, script) => {264 const text = $(script).text();265 if (text.includes("jsl.dh(")) {266 injectedHtml += text.replaceAll("\\x3c", "<").replaceAll("\\x3e", ">").replaceAll('\\"', '"');267 }268 });269 return cheerio2.load(injectedHtml);270}271function extractImageMap(html) {272 const imageMap = {};273 const ldiMatch = html.match(/google\.ldi\s*=\s*(\{.*?\});/s);274 if (ldiMatch) {275 try {276 Object.assign(imageMap, JSON.parse(ldiMatch[1]));277 } catch {278 }279 }280 const scriptPattern = /var\s+_u\s*=\s*'([^']+)'\s*;\s*var\s+_i\s*=\s*'([^']+)'\s*;\s*_setImagesSrc/g;281 let m;282 while ((m = scriptPattern.exec(html)) !== null) {283 const url = m[1].replaceAll("\\x3d", "=").replaceAll("\\x26", "&");284 imageMap[m[2]] = url;285 }286 return imageMap;287}288function isValidProductImage(url) {289 if (!url || !url.startsWith("http")) return false;290 return url.includes("shopping?q=tbn:") || url.includes("encrypted-tbn");291}292function extractMainImage($, imageMap) {293 const metaImg = $(SELECTORS.mainImageMeta).attr("content");294 if (metaImg && isValidProductImage(metaImg)) {295 return metaImg;296 }297 for (const url of Object.values(imageMap)) {298 if (isValidProductImage(url)) return url;299 }300 return null;301}302function extractAllImages($, mainImage) {303 const images = [];304 $(SELECTORS.galleryImage).each((_, el) => {305 const src = $(el).attr("data-src");306 if (isValidProductImage(src) && !images.includes(src)) {307 images.push(src);308 }309 });310 if (mainImage && isValidProductImage(mainImage)) {311 const existing = images.indexOf(mainImage);312 if (existing !== -1) images.splice(existing, 1);313 images.unshift(mainImage);314 }315 return images;316}317function extractRatingLabel($) {318 const el = $("span[aria-label]").filter((_, e) => ($(e).attr("aria-label") || "").includes("Rated")).first();319 return el.length ? el.attr("aria-label") : null;320}321function extractTitle($) {322 const titleEl = $(SELECTORS.title).first();323 if (titleEl.length) return nodeText(titleEl.get(0));324 const title = $("title").first().text();325 return title ? title.trim() : null;326}327function extractDescription($) {328 const container = $(SELECTORS.description).first();329 if (!container.length) return null;330 let textEl = container.find(SELECTORS.descriptionText).first();331 if (!textEl.length) textEl = container;332 return cleanText(nodeText(textEl.get(0), " "));333}334function extractRating(ratingLabel) {335 if (!ratingLabel) return null;336 const match = ratingLabel.match(/Rated\s+([\d.]+)/);337 return match ? parseFloat(match[1]) : null;338}339function extractReviewCount(ratingLabel) {340 if (!ratingLabel) return null;341 const match = ratingLabel.match(/([\d,]+\.?\d*[kKmM]?)\s+(?:user\s+)?reviews?/);342 if (!match) return null;343 const raw = match[1].replace(/,/g, "");344 const suffix = raw.slice(-1).toLowerCase();345 if (suffix === "k") return Math.trunc(parseFloat(raw.slice(0, -1)) * 1e3);346 if (suffix === "m") return Math.trunc(parseFloat(raw.slice(0, -1)) * 1e6);347 return Math.trunc(parseFloat(raw));348}349function extractSpecs($) {350 const specs = {};351 $(SELECTORS.spec).each((_, el) => {352 const text = nodeText(el, ":");353 const sep = text.indexOf(":");354 if (sep !== -1) {355 const key = cleanText(text.slice(0, sep));356 const value = cleanText(text.slice(sep + 1));357 if (key && value !== null) specs[key] = value;358 }359 });360 return specs;361}362function buildVariantUrl(baseUrl, pvf) {363 if (!baseUrl || !pvf) return null;364 let parsed;365 try {366 parsed = new URL(baseUrl);367 } catch {368 return null;369 }370 const prds = parsed.searchParams.get("prds");371 if (!prds) return null;372 const parts = prds.split(",").filter((p) => !p.startsWith("pvf:"));373 parts.push(`pvf:${pvf}`);374 parsed.searchParams.set("prds", parts.join(","));375 return parsed.toString();376}377function extractFilters($, $inj, baseUrl) {378 const filters = {};379 const seen = /* @__PURE__ */ new Set();380 const baseSelections = decodePvf(pvfFromUrl(baseUrl));381 const process = ($$) => {382 $$(SELECTORS.filterOption).each((_, el) => {383 const $el = $$(el);384 let category;385 const parentList = $el.closest(SELECTORS.filterList);386 if (parentList.length) {387 category = (parentList.attr("aria-label") || "").replace(" options", "").trim();388 } else {389 category = "Unknown";390 const prevHeading = findPrevious($$, el, SELECTORS.filterHeading);391 if (prevHeading) {392 category = nodeText(prevHeading, ":").split(":")[0].trim();393 }394 }395 const optionName = $el.attr("data-label") || cleanText(nodeText(el, " "));396 const optKey = `${category} ${optionName}`;397 if (seen.has(optKey)) return;398 seen.add(optKey);399 const ariaLabel = ($el.attr("aria-label") || "").toLowerCase();400 const isSelected = $el.attr("data-selected") === "true" || $el.attr("selected") === "true" || ariaLabel.includes("currently selected");401 if (!filters[category]) filters[category] = [];402 const opt = { name: optionName, selected: isSelected };403 const imgUrl = $el.attr("data-img");404 if (imgUrl) opt.image = imgUrl;405 if (category !== "Unknown" && optionName) {406 const pvf = encodePvf(withSelection(baseSelections, category, optionName));407 const variantUrl = buildVariantUrl(baseUrl, pvf);408 if (variantUrl) opt.url = variantUrl;409 }410 filters[category].push(opt);411 });412 };413 process($);414 process($inj);415 return Object.entries(filters).map(([category, options]) => ({ category, options }));416}417function extractCurrency(...values) {418 const text = values.filter(Boolean).join(" ");419 for (const [symbol, currency] of Object.entries(CURRENCY_BY_SYMBOL)) {420 if (text.includes(symbol)) return currency;421 }422 const match = text.match(/\b[A-Z]{3}\b/);423 return match ? match[0] : null;424}425function extractCurrentPrice($, cardNode) {426 const $card = $(cardNode);427 const isCurrent = (a) => a.startsWith("Current price:") || a.startsWith("Current price is");428 const $container = $card.find(SELECTORS.priceContainer).first();429 let $priceEl = $();430 if ($container.length) {431 $priceEl = findByAria($, $container, isCurrent);432 }433 if (!$priceEl.length) {434 $priceEl = findByAria($, $card, isCurrent);435 }436 if (!$priceEl.length && !$container.length) {437 return { price: null, price_label: null, currency: null };438 }439 const price = ($priceEl.length ? nodeText($priceEl.get(0)) : "") || ($container.length ? nodeText($container.get(0)) : "") || null;440 const priceLabel = $priceEl.length ? $priceEl.attr("aria-label") : null;441 const currency = ($container.length ? $container.attr("data-crcy") : null) || extractCurrency(price, priceLabel);442 return { price, price_label: priceLabel, currency };443}444function extractOldPrice($, cardNode) {445 const $oldPriceEl = findByAria($, $(cardNode), (a) => a.startsWith("Old price was") || a.startsWith("Maximum retail price:"));446 if (!$oldPriceEl.length) {447 return { old_price: null, old_price_label: null };448 }449 return {450 old_price: nodeText($oldPriceEl.get(0)) || null,451 old_price_label: $oldPriceEl.attr("aria-label")452 };453}454function extractOfferTitle($, cardNode) {455 const titleEl = $(cardNode).find(SELECTORS.offerTitle).first();456 if (!titleEl.length) return null;457 return cleanText(nodeText(titleEl.get(0), " "));458}459function extractOfferRating($, cardNode) {460 const ratingEl = findByAria($, $(cardNode), (a) => a.startsWith("Rated ") && a.includes(" out of 5"));461 if (!ratingEl.length) return null;462 return extractRating(ratingEl.attr("aria-label"));463}464function extractOfferStatus($, cardNode) {465 const statusEl = $(cardNode).find(SELECTORS.offerStatus).first();466 if (!statusEl.length) return null;467 return cleanText(nodeText(statusEl.get(0), " "));468}469function extractOfferDelivery($, cardNode) {470 const deliveryEl = findByAria($, $(cardNode), (a) => a.toLowerCase().includes("delivery"));471 if (!deliveryEl.length) return null;472 return cleanText(deliveryEl.attr("aria-label")) || cleanText(nodeText(deliveryEl.get(0), " "));473}474function extractSellersNew($, $cards) {475 const sellers = [];476 const seen = /* @__PURE__ */ new Set();477 $cards.each((_, cardNode) => {478 const $card = $(cardNode);479 const merchant = $card.find(SELECTORS.merchantContext).first().attr("data-report-feedback-about-context") || cleanText(nodeText($card.find(SELECTORS.merchantName).first().get(0), " "));480 const linkEl = $card.find(SELECTORS.externalLink).first();481 const price = extractCurrentPrice($, cardNode);482 const oldPrice = extractOldPrice($, cardNode);483 let offerRating = null;484 const ratingEl = $card.find(SELECTORS.offerRatingNew).first();485 if (ratingEl.length) {486 const m = ratingEl.text().match(/([\d.]+)/);487 if (m) offerRating = parseFloat(m[1]);488 }489 let sellerLogo = null;490 const src = $card.find("img").first().attr("src") || "";491 if (src.startsWith("http") && !src.includes("data:image")) sellerLogo = src;492 const seller = {493 merchant: merchant || null,494 merchant_id: null,495 offer_id: $card.attr("data-sori-id") || null,496 title: extractOfferTitle($, cardNode),497 price: price.price,498 currency: price.currency,499 old_price: oldPrice.old_price,500 target_url: linkEl.length ? linkEl.attr("href") : null,501 status: extractOfferStatus($, cardNode),502 delivery: extractOfferDelivery($, cardNode),503 offer_rating: offerRating,504 seller_logo: sellerLogo505 };506 const key = `${seller.merchant} ${seller.target_url}`;507 if (!seen.has(key)) {508 sellers.push(seller);509 seen.add(key);510 }511 });512 return sellers;513}514function extractSellersOld($, imageMap, offersGrid) {515 const sellers = [];516 const seen = /* @__PURE__ */ new Set();517 const $merchants = offersGrid.length ? offersGrid.find(SELECTORS.merchantOld) : $(SELECTORS.merchantOld);518 $merchants.each((_, el) => {519 const $el = $(el);520 let $card = $el.closest(SELECTORS.offerCardOld);521 if (!$card.length) $card = $el.parent();522 if (!$card.length) $card = $el;523 const cardNode = $card.first().get(0);524 const linkEl = $(cardNode).find("a[href]").first();525 const price = extractCurrentPrice($, cardNode);526 const oldPrice = extractOldPrice($, cardNode);527 let sellerLogo = null;528 const imgTag = $(cardNode).find("img").first();529 if (imgTag.length) {530 const imgId = imgTag.attr("id");531 if (imgId && imageMap[imgId] && imageMap[imgId].startsWith("http")) {532 sellerLogo = imageMap[imgId];533 } else {534 const src = imgTag.attr("src") || imgTag.attr("data-src") || "";535 if (src.startsWith("http") && !src.includes("data:image")) {536 sellerLogo = src;537 }538 }539 }540 const seller = {541 merchant: $el.attr("data-merchant-name") ?? null,542 merchant_id: $el.attr("data-merchantid") ?? null,543 offer_id: $el.attr("data-oid") ?? null,544 title: extractOfferTitle($, cardNode),545 price: price.price,546 currency: price.currency,547 old_price: oldPrice.old_price,548 target_url: $el.attr("data-target-url") || (linkEl.length ? linkEl.attr("href") : null),549 status: extractOfferStatus($, cardNode),550 delivery: extractOfferDelivery($, cardNode),551 offer_rating: extractOfferRating($, cardNode),552 seller_logo: sellerLogo553 };554 const key = `${seller.merchant} ${seller.merchant_id} ${seller.offer_id}`;555 if (!seen.has(key)) {556 sellers.push(seller);557 seen.add(key);558 }559 });560 return sellers;561}562function extractSellers($, imageMap) {563 const offersGrid = $(SELECTORS.offersGrid).first();564 const $newCards = offersGrid.length ? offersGrid.find(SELECTORS.offerCardNew) : $();565 if ($newCards.length) return extractSellersNew($, $newCards);566 return extractSellersOld($, imageMap, offersGrid);567}568function parseProduct(html, url, finalUrl) {569 const $ = cheerio2.load(html);570 const imageMap = extractImageMap(html);571 const $inj = extractInjectedSoup($);572 const mainImage = extractMainImage($, imageMap);573 const ratingLabel = extractRatingLabel($);574 const variantBaseUrl = finalUrl && finalUrl.includes("prds=") ? finalUrl : url;575 return {576 input_url: url,577 final_url: finalUrl,578 title: extractTitle($),579 description: extractDescription($),580 images: extractAllImages($, mainImage),581 rating: extractRating(ratingLabel),582 review_count: extractReviewCount(ratingLabel),583 features: extractSpecs($),584 filters: extractFilters($, $inj, variantBaseUrl),585 buying_options: extractSellers($, imageMap)586 };587}588
589// immersive/src/main.js590var OSHOP_BASE_URL = "https://www.google.com/search";591var GOOGLE_DOMAIN_RE = /(^|\.)(google\.[a-z]{2,}(\.\w{2})?)$/i;592var COUNTRY_CODE_RE = /^[a-z]{2}$/i;593function validateInput(url, country) {594 if (!url || !url.trim()) {595 throw new Error(596 'Input field "url" is required. Provide a Google Shopping product URL containing a prds= query parameter.'597 );598 }599 let parsed;600 try {601 parsed = new URL(url);602 } catch {603 throw new Error(604 `"url" does not look like a valid URL: ${JSON.stringify(url)}. Expected a full URL starting with https://www.google.com/...`605 );606 }607 const scheme = parsed.protocol.replace(/:$/, "");608 if (scheme.toLowerCase() !== "https") {609 throw new Error(610 `"url" must use the https scheme, got ${JSON.stringify(scheme)}. Google Shopping URLs always start with https://`611 );612 }613 if (!GOOGLE_DOMAIN_RE.test(parsed.hostname)) {614 throw new Error(615 `"url" must be a Google domain (e.g. google.com, google.co.in), got ${JSON.stringify(parsed.hostname)}.`616 );617 }618 if (!parsed.searchParams.get("prds")) {619 throw new Error(620 '"url" must contain a prds= query parameter. Open a product in Google Shopping, copy the full URL from the address bar \u2014 it should contain prds=eto:... or similar.'621 );622 }623 if (country && !COUNTRY_CODE_RE.test(country)) {624 throw new Error(625 `"country" must be a 2-letter ISO country code (e.g. "in", "us", "gb"), got ${JSON.stringify(country)}.`626 );627 }628}629function buildImmersiveUrl(url, country) {630 const parsed = new URL(url);631 const params = parsed.searchParams;632 if (!params.has("ibp")) params.set("ibp", "oshop");633 if (!params.has("hl")) params.set("hl", "en");634 if (!params.has("gl")) params.set("gl", country);635 if (!params.has("udm")) params.set("udm", "28");636 return `${OSHOP_BASE_URL}?${params.toString()}`;637}638async function runImmersive(httpClient, url, country) {639 const fetchUrl = buildImmersiveUrl(url, country);640 log2.info(`Fetching immersive product URL: ${fetchUrl}`);641 const [html, finalUrl] = await httpClient.fetch(fetchUrl);642 log2.info(`Fetched ${html.length} bytes of HTML. Final URL: ${finalUrl.slice(0, 100)}`);643 if (isBlocked(html)) {644 log2.warning("Blocked by Google (captcha / unusual traffic)");645 return null;646 }647 const product = parseProduct(html, url, finalUrl);648 log2.info(649 `Parsed product blocks: title=${product.title} features=${Object.keys(product.features || {}).length} buying_options=${(product.buying_options || []).length} filters=${(product.filters || []).length}`650 );651 if (Object.keys(product.features || {}).length === 0 && (product.buying_options || []).length === 0) {652 log2.warning(653 "Fetched HTML did not contain Google Shopping immersive product blocks. This usually means Google returned a sparse/static variant for this request."654 );655 await Actor.setValue("sparse-response.html", html, { contentType: "text/html" });656 }657 return product;658}659await Actor.init();660try {661 const input = await Actor.getInput() || {};662 const url = input.url;663 const country = input.country || "in";664 validateInput(url, country);665 const proxyConfiguration = await Actor.createProxyConfiguration({666 groups: ["RESIDENTIAL"],667 countryCode: country.toUpperCase()668 });669 const proxyUrl = await proxyConfiguration.newUrl();670 const httpClient = new ProxyHttpClient(proxyUrl);671 const product = await runImmersive(httpClient, url, country);672 if (product) {673 await Actor.pushData(product);674 log2.info("Pushed immersive product details to dataset");675 }676} finally {677 await Actor.exit();678}{ "name": "@gs/immersive", "version": "0.0.1", "type": "module", "dependencies": { "apify": "^3.2.6", "cheerio": "^1.0.0", "got-scraping": "^4.0.6" }}