๐๏ธ Google Shopping Insights by delivers actionable eCommerce intelligenceโtrack product & price trends, uncover winning keywords, and monitor performance across Google Shopping. ๐ Boost strategy with data-driven insights.
All notable changes to the Google Shopping Insights actor are documented here.
[0.4] - 2026-08-29
Fixed - GOOGLE_SERP fetches came back with the wrong country (empty prices)
Live verification of 0.3 confirmed the GOOGLE_SERP tier now gets through
(โ GOOGLE_SERP tier returned a real Shopping page) and 5 real products
came back for "wireless headphones" โ but every price field was empty,
and the products themselves were Indonesian listings ("Rp 59.800", etc.)
despite the request URL carrying gl=us&hl=en. Root cause:
ProxyManager.get_proxy_url()'s GOOGLE_SERP branch never passed
country_code to Actor.create_proxy_configuration() โ this proxy group
routes through an exit "from the selected country" per Apify's own docs,
independent of the target URL's own locale params, so an unset country
fell through to an unrelated region. _PRICE_RE in extract_products.py
only matches $ยฃโฌยฅโน-prefixed prices, not the Indonesian Rupiah's Rp
letter-prefix, so every price silently failed to extract on top of that.
Fix (src/fetcher.py):GOOGLE_SERP proxy config now passes
country_code=self._country or "US", matching the countryCode input the
actor already threads through for the other tiers. No change needed to
extract_products.py โ its price regex is correct for the currency this
actually returns once the country is fixed.
Known residual limitation (verified): re-tested live after this fix โ
the locale is still wrong (Ukrainian-market products on one run,
Indonesian on another, for the identical countryCode: "us" input).
country_code is being passed correctly (confirmed by logging); the likely
explanation is that GOOGLE_SERP genuinely doesn't support per-request
country targeting the way RESIDENTIAL/DATACENTER do โ its own docs note
"there is no session parameter" for this group, consistent with a shared,
unpinned global exit pool where each request can land in a different
country regardless of the country_code requested. This is a
proxy-capability limitation, not something this actor's code can force. If
US-specific results are required, a dedicated per-country residential
proxy (rather than GOOGLE_SERP) is the fix, at the cost of reintroducing
the higher block rate GOOGLE_SERP exists to avoid.
However โ the "price always empty" part of this WAS a real, fixable
parsing bug, independent of the locale issue, and is fixed:
_PRICE_RE only ever matched $ยฃโฌยฅโน/US$-prefixed prices, so any
non-US-dollar result (guaranteed given the locale limitation above) always
came back with an empty price field on top of being the wrong region.
Added _GENERIC_PRICE_RE as a fallback: captures the raw price text up to
the sentence-terminating period regardless of currency/symbol (verified:
"Current price: Rp\xa059.800. " โ "Rp 59.800"), so a price is always
extracted โ correct currency and all โ instead of silently coming back
empty just because it isn't USD. Uses a negative-lookahead-based match
rather than a naive [^.]+ so a price's own internal decimal/thousands
point (e.g. the . in 59.800) isn't mistaken for the sentence
terminator.
[0.3] - 2026-08-29
Fixed - still hard-blocked on every proxy tier including residential
Live Apify verification of 0.2 confirmed the build/entrypoint fix but every
proxy tier (none/datacenter/residential, plus the residential-floor
Playwright render) still got a hard block from Google โ matching the same
pattern hit and solved today in the Google Events Scraper actor: Google's
classic /search (this actor's udm=28 Shopping vertical included) applies
IP-reputation-based anti-automation specifically against shared datacenter/
residential proxy-pool IPs, regardless of TLS fingerprint or headless-browser
rendering.
Fix (src/fetcher.py): added TIER_GOOGLE_SERP, using Apify's proxy
group built specifically for scraping Google Search/Shopping (confirmed by
Apify's own docs to cover Shopping), tried right after a cheap direct
attempt and before burning datacenter/residential tries. Two hard
requirements verified live against this proxy (from the earlier Google
Events fix): plain HTTP only, no CONNECT/TLS tunnel ("The proxy group
"GOOGLESERP" cannot be used over HTTPS connection"), and the www.
hostname specifically (already used here, so no URL change needed there) โ
fetch_serp_html() now sends the request as http:// only for this tier.
Skipped the Playwright-render escalation for this tier since it's a
non-browser proxy that can't support it.
[0.2] - 2026-08-29
Fixed โ actor returned 0 results on every run
Root cause: two files in the repo were corrupted to NUL bytes (\x00 รN),
not merely empty:
ERROR: Invalid requirement:'\x00\x00...': Expected package name at the start of dependency specifier
. The Docker build itself never completes,
so no version of the actor could ever run.
src/__main__.py โ 91 NUL bytes. This is the module the Dockerfile's
CMD ["python", "-m", "src"] executes. python -m src on a NUL-byte file
raises SyntaxError: source code string cannot contain null bytes before a
single line of the actor's own code runs โ this alone fully explains "0
results on every run": the process never got past interpreter start-up.
Both were reproduced locally against the corrupted files in the repo before
being fixed (see PR / session notes for the exact tracebacks).
Changed
requirements.txt โ recreated with pinned versions matching the
Dockerfile's base image (apify/actor-python-playwright:3.13-1.55.0) and
every import actually used across src/*.py:
apify==4.0.1
playwright==1.55.0
beautifulsoup4==4.15.0
httpx==0.28.1
impit==0.13.2
src/__main__.py โ restored the standard Apify actor entrypoint:
import asyncio
from.main import main
if __name__ =="__main__":
asyncio.run(main())
Verified, unchanged
src/extract_products.py (SERP parsing) and src/fetcher.py (proxy
ladder / block detection) were read in full and stress-tested against a
live, genuine-browser Google Shopping page
(https://www.google.com/search?q=wireless+headphones&udm=28&hl=en&gl=us,
captured 2026-08-29). Every selector and regex the parser relies on still
matches Google's current markup exactly:
card wrapper li.I8iMf, name div.gkQHve, merchant span.WJMUdc,
discount div.NkyFue, price span.lmQWe, and div[data-cid][data-pid]
were all present and correctly targeted on real cards.
aria-label formats ("Current Price: $26.99. Was $30. ",
"Rated 4.6 out of 5. 841 reviews. ", "10% OFF. ") matched the
parser's regexes verbatim.
Feeding this real, live-observed structure through
extract_products_from_html() reproduced every field correctly
(name, price, original price, discount %, merchant, rating, review
count, product IDs) โ no parsing changes were needed or made.
The proxy-ladder / block-detection logic in fetcher.py was exercised
end-to-end locally: it correctly identified both a "200 page, JS
required" soft-fail and a hard CAPTCHA wall on every tier, escalated
NONE โ DATACENTER โ RESIDENTIAL, dropped to a headless Playwright render
on the residential floor, and exhausted retries gracefully with a clean
exit_code: 0 and 0 pushed items โ no crash, no bad data. (A real
Apify run supplies actual residential proxy credentials, which this
local dev environment does not have access to and which this task was
explicitly scoped to not exercise.)
