# 🛍️ Google Shopping Insights (`scrapier/google-shopping-insights`) Actor Extract product insights from Google Shopping, including pricing, sellers, ratings, reviews, availability, and product details. Monitor competitors, analyze market trends, track pricing changes, and gather e-commerce intelligence with structured data at scale. - **URL**: https://apify.com/scrapier/google-shopping-insights.md - **Developed by:** [Scrapier](https://apify.com/scrapier) (community) - **Categories:** E-commerce, AI, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $4.99 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## 🛍️ Google Shopping Insights Extract rich, structured product data from **Google Shopping** (`udm=28`) at scale — product names, prices, discounts, merchants, conditions, review scores, images and stable Google product IDs — with a built-in **smart proxy fallback** that adapts to Google's block walls so you don't have to. --- ### 🌟 Why choose this Actor? - 🛡️ **Smart 3-tier proxy fallback** — starts **direct (no proxy)**, escalates to **datacenter**, then **residential** (3 retries). One block does not kill the run. - 🧠 **Browser-grade requests** — uses `impit` browser-impersonation (real TLS + HTTP/2 fingerprints) and escalates to a headless **Playwright** render on the toughest pages. - 🔴 **Live streaming** — every product is pushed to the dataset the moment it is parsed, so a crash mid-run still leaves you with the rows already scraped. - 🧩 **Resilient parsing** — class-based fast path with a structural fallback for *every* field, so it survives Google's constantly-rotating CSS class names. - 🔍🌐 **Two ways to query** — paste search keywords **or** ready-made Google Shopping URLs (both bulk). --- ### 🔑 Key features | Feature | Description | | --- | --- | | 🔍 Search queries | Bulk list of keywords — each becomes its own output section | | 🌐 Bulk URLs | Paste many Google Shopping URLs at once | | 🌎 Country targeting | 62 countries for the Google `gl` geo parameter | | 💰 Comparison prices | Original price + discount percentage per product | | 📣 Sponsored toggle | Keep or drop sponsored cards | | 🎁 Hard caps | Limit results in total and per section | | 🛡️ Proxy fallback | Auto-escalates on block, then locks to the first tier that works | | 📊 Two dataset views | Overview + Full Details, both with emoji column labels | --- ### 📥 Input ```json { "queries": ["iPhone"], "startUrls": ["https://www.google.com/search?q=android&udm=28&tbs=vw:l"], "countryCode": "us", "includeComparisonPrices": true, "includeExtraProductDetails": true, "includeSponsoredResults": true, "maxItems": 60, "maxItemsPerQuery": 60, "requestDelay": 1, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ```` **Field reference** - **`queries`** — list of product search terms. Each query is scraped as its own section. Takes precedence over `startUrls`. - **`startUrls`** — list of ready-made Google Shopping result URLs (the `udm=28` / `tbm=shop` pages). - **`countryCode`** — country used for Google's `gl` parameter and proxy geo. Applies to **queries** only. - **`includeComparisonPrices`** — include original price + discount percent (turns on extra details automatically). - **`includeExtraProductDetails`** — include the product spec / sub-title line. - **`includeSponsoredResults`** — keep sponsored product cards. - **`maxItems`** — hard cap on total products pushed (leave empty for no limit). - **`maxItemsPerQuery`** — hard cap per query/URL. - **`requestDelay`** — polite base delay (seconds) between page requests; jitter is added automatically. - **`proxyConfiguration`** — Apify proxy editor. The actor decides the fallback ladder on its own; you do not need to change this. *** ### 📤 Output Each pushed dataset row: ```json { "productName": "Apple iPhone 17 Pro", "productLink": "https://www.google.com/search?q=iPhone&tbs=vw:l&start=0&gl=US&hl=en&udm=28&sei=...#pvs=0&rank=1", "price": "$62.45/mo", "withoutDiscountPrice": "$62.45/mo", "discountPercent": "", "merchantName": "Apple", "productCondition": "", "reviewsScore": "4.5", "reviewsCount": "13K", "productImage": "https://encrypted-tbn...", "positionOnSearchPage": 1, "productDetails": "", "isSponsored": false, "productIds": { "cid": "461677229171638444", "gid": "...", "iid": "...", "oid": "...", "pid": "..." }, "searchKeyword": "iPhone" } ``` The **`searchKeyword`** field tells you which query/section each product belongs to, so a single dataset cleanly separates results per section. *** ### 🚀 How to use (Apify Console) 1. Log in at → **Actors**. 2. Open this actor (`google-shopping-insights`). 3. Add one or more **Search Queries** 🔍 and/or **Google Shopping URLs** 🌐. 4. Pick a **Country** 🌎 and tweak the data options and limits. 5. Click **Start**. 6. Watch the **Logs** tab — every product is announced in real time. 7. Open **Storage → Dataset** to inspect the **Overview** and **Full Details** views. 8. Export to JSON / CSV / Excel / XML / HTML. *** ### 🤖 Use via API ```bash curl -X POST "https://api.apify.com/v2/acts/~google-shopping-insights/runs?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "queries": ["iPhone"], "countryCode": "us", "maxItems": 60 }' ``` *** ### 🛡️ Smart proxy fallback ``` DIRECT ─► DATACENTER ─► RESIDENTIAL (×3 retries) ``` - The run starts **without a proxy** and requests Google directly. - On a Google block (HTTP 403/429/503, captcha, "unusual traffic", empty body), the actor switches to **DATACENTER**. - If that also blocks, it switches to **RESIDENTIAL** and retries up to **3 times** with fresh residential sessions (escalating to a real headless browser render as a last resort). - Once a tier succeeds, the actor **locks** to that tier for the rest of the run. - Every transition is logged with a clear 🛑 / 🔒 / 🌐 marker. *** ### ⭐ Best use cases - 📈 Price monitoring and competitive intelligence - 🛒 Building product / price-comparison datasets - 🏪 Merchant and availability tracking - 🤖 Feeding AI workflows that need fresh retail data *** ### 💰 Pricing **Pay-per-result.** Billed per product row pushed to the default dataset via Apify's synthetic `apify-default-dataset-item` event. No charge for failed runs, no fixed monthly fee. The Store page shows the live per-item price. *** ### ❓ FAQ **Do I need a proxy?** No. The actor tries direct first and only spins up paid proxies if Google blocks the direct request. **Why are some fields empty?** Google does not render every chip (discount, condition, reviews) on every card. The parser falls back to structural selectors, but if Google omits the data there is nothing to recover. **Can I scrape multiple keywords at once?** Yes — add several **Search Queries**; each is scraped as its own section, tagged by `searchKeyword`. *** ### ⚖️ Legal & ethical - Data is collected only from **publicly available** Google Shopping search results. - You are responsible for compliance with GDPR / CCPA, anti-spam laws, and Google's Terms of Service. - Respect target sites' rate limits and `robots.txt`. *** ### 📬 Support & feedback Found a bug or want a new feature? Reach out via the Apify Console. # Actor input Schema ## `queries` (type: `array`): One or more product search terms (e.g. iPhone, gaming laptop). Each query becomes its own section in the output. Takes precedence over Start URLs. ## `startUrls` (type: `array`): Paste one or more Google Shopping result URLs (the udm=28 / tbm=shop pages). Mix multiple URLs to scrape several listings in a single run. Country only applies to queries, not URLs. ## `countryCode` (type: `string`): Country used for Google's geo (gl) parameter and proxy IP. Applies to Search Queries only. ## `includeComparisonPrices` (type: `boolean`): Include the original (pre-discount) price and the discount percentage for each product. Slightly increases resource usage. ## `includeExtraProductDetails` (type: `boolean`): Include the extra product spec / sub-title line. Ignored (always on) when 'Include comparison prices' is enabled. ## `includeSponsoredResults` (type: `boolean`): If enabled, sponsored product cards are also included in the results. ## `maxItems` (type: `integer`): Hard cap on the total number of products pushed across all sections. Leave empty for no limit. ## `maxItemsPerQuery` (type: `integer`): Hard cap on products collected per query/URL. Leave empty to use the total limit. ## `proxyConfiguration` (type: `object`): The actor starts with NO proxy, then auto-escalates to DATACENTER and finally RESIDENTIAL (×3 retries) if Google blocks. You don't need to change anything here. ## Actor input object example ```json { "queries": [ "iPhone" ], "startUrls": [ "https://www.google.com/search?q=android&udm=28&tbs=vw:l" ], "countryCode": "us", "includeComparisonPrices": true, "includeExtraProductDetails": true, "includeSponsoredResults": true, "maxItems": 10, "maxItemsPerQuery": 10, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "queries": [ "iPhone" ], "startUrls": [ "https://www.google.com/search?q=android&udm=28&tbs=vw:l" ], "maxItems": 10, "maxItemsPerQuery": 10, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("scrapier/google-shopping-insights").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "queries": ["iPhone"], "startUrls": ["https://www.google.com/search?q=android&udm=28&tbs=vw:l"], "maxItems": 10, "maxItemsPerQuery": 10, "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], }, } # Run the Actor and wait for it to finish run = client.actor("scrapier/google-shopping-insights").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "queries": [ "iPhone" ], "startUrls": [ "https://www.google.com/search?q=android&udm=28&tbs=vw:l" ], "maxItems": 10, "maxItemsPerQuery": 10, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }' | apify call scrapier/google-shopping-insights --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapier/google-shopping-insights", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "🛍️ Google Shopping Insights", "description": "Extract product insights from Google Shopping, including pricing, sellers, ratings, reviews, availability, and product details. Monitor competitors, analyze market trends, track pricing changes, and gather e-commerce intelligence with structured data at scale.", "version": "0.1", "x-build-id": "1XOcNJnFihmSEbZpM" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapier~google-shopping-insights/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapier-google-shopping-insights", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapier~google-shopping-insights/runs": { "post": { "operationId": "runs-sync-scrapier-google-shopping-insights", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapier~google-shopping-insights/run-sync": { "post": { "operationId": "run-sync-scrapier-google-shopping-insights", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "queries": { "title": "🔍 Search Queries (bulk supported)", "type": "array", "description": "One or more product search terms (e.g. iPhone, gaming laptop). Each query becomes its own section in the output. Takes precedence over Start URLs.", "items": { "type": "string" } }, "startUrls": { "title": "🌐 Google Shopping URLs (bulk supported)", "type": "array", "description": "Paste one or more Google Shopping result URLs (the udm=28 / tbm=shop pages). Mix multiple URLs to scrape several listings in a single run. Country only applies to queries, not URLs.", "items": { "type": "string" } }, "countryCode": { "title": "🌎 Country", "enum": [ "us", "gb", "ca", "au", "nz", "ie", "de", "fr", "es", "it", "pt", "nl", "be", "ch", "at", "se", "no", "dk", "fi", "pl", "cz", "gr", "ro", "hu", "ua", "tr", "ru", "in", "pk", "bd", "lk", "np", "cn", "hk", "tw", "jp", "kr", "sg", "my", "id", "th", "vn", "ph", "ae", "sa", "qa", "kw", "bh", "om", "il", "eg", "za", "ng", "ke", "gh", "ma", "br", "mx", "ar", "cl", "co", "pe" ], "type": "string", "description": "Country used for Google's geo (gl) parameter and proxy IP. Applies to Search Queries only.", "default": "us" }, "includeComparisonPrices": { "title": "💰 Include comparison prices", "type": "boolean", "description": "Include the original (pre-discount) price and the discount percentage for each product. Slightly increases resource usage.", "default": true }, "includeExtraProductDetails": { "title": "📋 Include extra product details", "type": "boolean", "description": "Include the extra product spec / sub-title line. Ignored (always on) when 'Include comparison prices' is enabled.", "default": true }, "includeSponsoredResults": { "title": "📣 Include sponsored products", "type": "boolean", "description": "If enabled, sponsored product cards are also included in the results.", "default": true }, "maxItems": { "title": "🎁 Maximum results (total)", "minimum": 1, "type": "integer", "description": "Hard cap on the total number of products pushed across all sections. Leave empty for no limit." }, "maxItemsPerQuery": { "title": "🔢 Maximum results per section", "minimum": 1, "type": "integer", "description": "Hard cap on products collected per query/URL. Leave empty to use the total limit." }, "proxyConfiguration": { "title": "🛡️ Proxy configuration", "type": "object", "description": "The actor starts with NO proxy, then auto-escalates to DATACENTER and finally RESIDENTIAL (×3 retries) if Google blocks. You don't need to change anything here.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```