🛍️ Google Shopping Insights

Extract rich, structured product data from Google Shopping (udm=28) at scale — product names, prices, discounts, merchants, conditions, review scores, images and stable Google product IDs — with a built-in smart proxy fallback that adapts to Google's block walls so you don't have to.

---

🌟 Why choose this Actor?

• 🛡️ Smart 3-tier proxy fallback — starts direct (no proxy), escalates to datacenter, then residential (3 retries). One block does not kill the run.
• 🧠 Browser-grade requests — uses impit browser-impersonation (real TLS + HTTP/2 fingerprints) and escalates to a headless Playwright render on the toughest pages.
• 🔴 Live streaming — every product is pushed to the dataset the moment it is parsed, so a crash mid-run still leaves you with the rows already scraped.
• 🧩 Resilient parsing — class-based fast path with a structural fallback for every field, so it survives Google's constantly-rotating CSS class names.
• 🔍🌐 Two ways to query — paste search keywords or ready-made Google Shopping URLs (both bulk).

---

🔑 Key features

Feature	Description
🔍 Search queries	Bulk list of keywords — each becomes its own output section
🌐 Bulk URLs	Paste many Google Shopping URLs at once
🌎 Country targeting	62 countries for the Google gl geo parameter
💰 Comparison prices	Original price + discount percentage per product
📣 Sponsored toggle	Keep or drop sponsored cards
🎁 Hard caps	Limit results in total and per section
🛡️ Proxy fallback	Auto-escalates on block, then locks to the first tier that works
📊 Two dataset views	Overview + Full Details, both with emoji column labels

---

📥 Input

{
  "queries": ["iPhone"],
  "startUrls": ["https://www.google.com/search?q=android&udm=28&tbs=vw:l"],
  "countryCode": "us",
  "includeComparisonPrices": true,
  "includeExtraProductDetails": true,
  "includeSponsoredResults": true,
  "maxItems": 60,
  "maxItemsPerQuery": 60,
  "requestDelay": 1,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

Field reference

• queries — list of product search terms. Each query is scraped as its own section. Takes precedence over startUrls.
• startUrls — list of ready-made Google Shopping result URLs (the udm=28 / tbm=shop pages).
• countryCode — country used for Google's gl parameter and proxy geo. Applies to queries only.
• includeComparisonPrices — include original price + discount percent (turns on extra details automatically).
• includeExtraProductDetails — include the product spec / sub-title line.
• includeSponsoredResults — keep sponsored product cards.
• maxItems — hard cap on total products pushed (leave empty for no limit).
• maxItemsPerQuery — hard cap per query/URL.
• requestDelay — polite base delay (seconds) between page requests; jitter is added automatically.
• proxyConfiguration — Apify proxy editor. The actor decides the fallback ladder on its own; you do not need to change this.

---

📤 Output

Each pushed dataset row:

{
  "productName": "Apple iPhone 17 Pro",
  "productLink": "https://www.google.com/search?q=iPhone&tbs=vw:l&start=0&gl=US&hl=en&udm=28&sei=...#pvs=0&rank=1",
  "price": "$62.45/mo",
  "withoutDiscountPrice": "$62.45/mo",
  "discountPercent": "",
  "merchantName": "Apple",
  "productCondition": "",
  "reviewsScore": "4.5",
  "reviewsCount": "13K",
  "productImage": "https://encrypted-tbn...",
  "positionOnSearchPage": 1,
  "productDetails": "",
  "isSponsored": false,
  "productIds": { "cid": "461677229171638444", "gid": "...", "iid": "...", "oid": "...", "pid": "..." },
  "searchKeyword": "iPhone"
}

The searchKeyword field tells you which query/section each product belongs to, so a single dataset cleanly separates results per section.

---

🚀 How to use (Apify Console)

1. Log in at https://console.apify.com → Actors.
2. Open this actor (google-shopping-insights).
3. Add one or more Search Queries 🔍 and/or Google Shopping URLs 🌐.
4. Pick a Country 🌎 and tweak the data options and limits.
5. Click Start.
6. Watch the Logs tab — every product is announced in real time.
7. Open Storage → Dataset to inspect the Overview and Full Details views.
8. Export to JSON / CSV / Excel / XML / HTML.

---

🤖 Use via API

curl -X POST "https://api.apify.com/v2/acts/<USER>~google-shopping-insights/runs?token=$APIFY_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{ "queries": ["iPhone"], "countryCode": "us", "maxItems": 60 }'

---

🛡️ Smart proxy fallback

DIRECT  ─►  DATACENTER  ─►  RESIDENTIAL (×3 retries)

• The run starts without a proxy and requests Google directly.
• On a Google block (HTTP 403/429/503, captcha, "unusual traffic", empty body), the actor switches to DATACENTER.
• If that also blocks, it switches to RESIDENTIAL and retries up to 3 times with fresh residential sessions (escalating to a real headless browser render as a last resort).
• Once a tier succeeds, the actor locks to that tier for the rest of the run.
• Every transition is logged with a clear 🛑 / 🔒 / 🌐 marker.

---

⭐ Best use cases

• 📈 Price monitoring and competitive intelligence
• 🛒 Building product / price-comparison datasets
• 🏪 Merchant and availability tracking
• 🤖 Feeding AI workflows that need fresh retail data

---

💰 Pricing

Pay-per-result. Billed per product row pushed to the default dataset via Apify's synthetic apify-default-dataset-item event. No charge for failed runs, no fixed monthly fee. The Store page shows the live per-item price.

---

❓ FAQ

Do I need a proxy? No. The actor tries direct first and only spins up paid proxies if Google blocks the direct request.

Why are some fields empty? Google does not render every chip (discount, condition, reviews) on every card. The parser falls back to structural selectors, but if Google omits the data there is nothing to recover.

Can I scrape multiple keywords at once? Yes — add several Search Queries; each is scraped as its own section, tagged by searchKeyword.

---

⚖️ Legal & ethical

• Data is collected only from publicly available Google Shopping search results.
• You are responsible for compliance with GDPR / CCPA, anti-spam laws, and Google's Terms of Service.
• Respect target sites' rate limits and robots.txt.

---

📬 Support & feedback

Found a bug or want a new feature? Reach out via the Apify Console.