Google Shopping Scraper

A small, read-only Google Shopping scraper that replays the public product-search document with HTTPCloak and parses server-rendered product cards. It does not use browser cookies, accounts, CAPTCHA solving, private RPCs, or mutation endpoints.

Search by image

In addition to text queries, the Actor accepts product photos and runs one of two modes (imageSearchMode):

Mode	Behavior
identifyAndShop (default)	Upload to Google Lens, derive a query from OCR, fallbackText, a descriptive filename, or Lens results, then scrape Google Shopping
lensProducts	Upload to Google Lens and return Products-tab cards using the Lens parser or the standard Shopping parser, depending on Google's markup

Image inputs (same contract as the Google Lens Actor):

• imageUrl / imageUrls — public HTTP(S) image URLs
• imagesBase64 — raw base64 or data:image/...;base64,... URIs
• imageKvsRecords — Apify key-value store binary records

Text queries and images can be combined in one run. Dataset rows from image searches set query to the image source (https://…, image:base64:1, or image:kvs:…), plus searchMode and (for identify) identifiedQuery.

Example Actor input:

{
  "imageUrl": "https://example.com/eufy-solocam-s340.jpg",
  "imageSearchMode": "identifyAndShop",
  "country": "US",
  "language": "en",
  "maxResults": 40
}

When OCR finds no usable text, provide fallbackText or use a descriptive filename such as koss-headphones.jpg. Generic names such as IMG_1234.jpg cannot provide a safe query fallback.

Image upload uses Apify RESIDENTIAL proxy; Shopping / Lens result fetches use GOOGLE_SERP.

Install

python3 -m venv .venv
.venv/bin/pip install -e .

Apify Actor

The repository is deployable as a Python Actor. Actor input accepts one or more queries, stores product rows in the default dataset, and writes sanitized run metrics to the SUMMARY key-value-store record.

The Actor creates proxy URLs through the Apify SDK at runtime. It uses the GOOGLE_SERP group for the SERP route and RESIDENTIAL with the requested country for the residential route. Proxy passwords must not be added to Actor input or source files.

Concurrency

The Actor processes independent searches concurrently. maxConcurrency defaults to 8, can be set from 1 to 16, and is the global/text-search limit. Google Lens work is separately capped at 3 because uploads and visual result parsing are heavier. Text and image jobs are interleaved so a batch of images cannot starve keyword searches.

Every concurrent job receives its own pool of three sticky residential session IDs. Dataset writes are serialized after the scrape slot is released, so slow storage writes neither corrupt a batch nor consume search concurrency. If Google challenges rise under load, lower maxConcurrency without rebuilding the Actor.

apify validate-schema
apify push

Proxy configuration

Proxy URLs are loaded only from environment variables. Do not put credentials in source code, command-line arguments, logs, or committed .env files.

export GOOGLE_SHOPPING_RESIDENTIAL_PROXY='http://user:password@host:port'
export GOOGLE_SHOPPING_SERP_PROXY='http://user:password@host:port'

When both variables are set, --route auto attempts the SERP route first. The residential route is used after a connection failure, timeout, proxy-authentication failure, transient 408, 425, 502, 503, or 504 response. A challenge on the first SERP Shopping request opens one fresh SERP session and retries once; if that retry is also challenged, auto mode tries the residential route. On Apify, an initial challenge can rotate through up to three isolated residential sessions; each has a unique sticky-session ID, and a challenged identity is discarded rather than reused. Use --route serp or --route residential in the standalone CLI to prevent cross-route fallback and make the cost model explicit.

Residential and direct routes perform a same-session warm-up through the selected route by loading https://www.google.com/?hl=.... The SERP route does not discard a paid Shopping response as a warm-up: its first valid Shopping response is parsed directly, so the normal SERP path costs exactly one proxy request. The supplied SERP proxy rejects HTTPS CONNECT requests but supports Google through a plain-HTTP target URL, so only that specialized route uses http://www.google.com/search; residential and direct routes remain HTTPS. The SERP route uses curl over HTTP/1.1 because this proxy fails before receiving an HTTP status with HTTPCloak; the other routes use HTTPCloak over HTTP/2. Both transports keep any same-route cookies in memory. Cookie values are never printed, saved, or included in output. Use --warmup-delay to control the bounded pause between the residential/direct warm-up and search requests.

A residential HEAD request was tested as a low-traffic bootstrap and returned Google cookies without a response body. Those cookies were all marked Secure. They are deliberately not forwarded to the SERP route because its required plain-HTTP target cannot carry Secure cookies with browser-correct semantics.

Debug metrics separate serp_request_count from residential_request_count. At a SERP price of $0.003 per request, the normal single-request path has a $0.003 proxy cost per query before Actor/platform fees. response_bytes is decoded HTML size and must not be treated as residential billable wire traffic.

Pagination

The scraper follows the browser-observed More results link returned in each Shopping document. It does not guess offsets: the next Google /search URL, including its start value and page-scoped query values, must be present in the current response. Pagination is bounded by both --max-pages and --max-results, deduplicates Google data-pid values across pages, and stops on a missing/repeated next link, no new products, an access challenge, or schema drift.

.venv/bin/google-shopping-scraper \
  "wireless headphones" \
  --route serp \
  --max-pages 2 \
  --max-results 100 \
  --debug

Every page in a pagination chain stays on the same transport and proxy session. SERP page 1 therefore leads to SERP page 2; a residential chain must start and continue residentially. TLS state cannot be moved between curl and HTTPCloak, and the client deliberately does not copy cookies or page tokens across proxy identities after an access challenge.

The bounded browser pilot for wireless headphones observed 53 parser-compatible products on page 1 and 40 on page 2, with three overlapping Google product IDs: 90 unique products across two requests. The saved raw one-request SERP document now yields 51 unique products with the container-aware parser. These counts are observations, not guaranteed page sizes.

The client deliberately does not change identities after 401, 403, 429, redirects, unexpected content types, product schema drift, or a challenge after the first successful page. Those outcomes stop the run. Before any Shopping page succeeds, auto mode may retry one fresh SERP transport and then the Actor's bounded pool of isolated residential sessions.

Run

Write JSONL to stdout:

.venv/bin/google-shopping-scraper \
  "wireless headphones" \
  --route serp \
  --country US \
  --language en \
  --max-results 20

Write a file and print sanitized metrics:

.venv/bin/google-shopping-scraper \
  "wireless headphones" \
  --max-results 20 \
  --output results.jsonl \
  --debug

Each row contains the title, displayed price, seller, optional original price, discount, delivery/returns text, rating, review count, image URL, and Google card identifiers observed in the result.

Tests

$.venv/bin/python -m unittest discover -s tests -v

The tests cover browser-observed card parsing, stable-ID deduplication, result and page limits, next-link pagination, the single-request SERP path, residential warm-up/search behavior, SERP-to-residential transient fallback, fresh residential-session rotation before the first successful page, and fail-closed handling for 403, post-page challenges, schema drift, global concurrency, lane limits, stable result order, and non-blocking persistence handoff.

Investigation contract

• Goal: retrieve product-search records from public Google Shopping results.
• Surface: GET https://www.google.com/search with q, hl, gl, udm=28, and shopmd=1.
• Identity: anonymous public page; no account or session data.
• Methods: one product-search request per followed page on the SERP route; residential/direct use one read-only warm-up followed by bounded product-search pages in the same session; no detail-page fanout.
• Default bound: at most 20 records and one result page; one request on SERP or two on residential/direct at that default.
• Stop conditions: access boundary, challenge, persistent/transient route exhaustion, unexpected content type/status, or unrecognized product markup.

Browser observation on July 27, 2026 confirmed product cards exposed as role="button" nodes whose accessible label contains Current Price: and Go to product viewer for this item. Stable card IDs were available from the nearest ancestor's data-pid; fields were mapped from descendants and are covered by the fixture tests.