Google Shopping Results Scraper

An Apify Actor written in Python that scrapes product details from Google Shopping based on search queries. It extracts essential product IDs (such as Catalog ID, GPC ID, headline offer ID, and image ID) and constructs direct Google Shopping detail URLs.

Features

• Google SERP Proxy Integration: Employs Apify's Google SERP proxy groups to bypass search engine blocks.
• Detailed Product Identifiers: Extracts:
  • Catalog ID (catalogid): Google's identifier for the product catalog entry.
  • GPC ID (gpcid): Global Product Code.
  • Headline Offer Doc ID (headline_offer_docid): The ID of the primary/headline merchant offer.
  • Image Doc ID (image_docid): The ID for the main product image.
• Direct Shopping Links: Synthesizes a valid https://www.google.com/search URL that opens the Google Shopping product comparison page directly (ibp=oshop).
• Robust Parsing: Built with aiohttp for async HTTP requests and BeautifulSoup4 for HTML structure extraction.

Input Configuration

The Actor accepts input in JSON format. The schema is defined in .actor/input_schema.json.

Field	Type	Required	Default	Description
q	String	Yes	-	The product search query (e.g., wireless headphones).
country	String	No	us	The Google country code for localized results (e.g., us, gb, in, ca).

Input Example

{
  "q": "mechanical keyboard",
  "country": "us"
}

Output Data Structure

The results are stored in the default dataset. For each product found, the output contains the following fields:

• title: The product card title.
• catalogid: Google's internal product catalog ID.
• gpcid: Global Product Code ID.
• headline_offer_docid: Document ID of the merchant offer shown.
• image_docid: Document ID of the product image.
• url: Direct link to the Google Shopping comparison interface.

Output Example

{
  "title": "Keychron K2 Wireless Mechanical Keyboard",
  "catalogid": "1234567890123456789",
  "gpcid": "9876543210",
  "headline_offer_docid": "off_abc123xyz",
  "image_docid": "img_789qwerty",
  "url": "https://www.google.com/search?ibp=oshop&q=mechanical+keyboard&prds=catalogid%3A1234567890123456789%2CheadlineOfferDocid%3Aoff_abc123xyz%2CimageDocid%3Aimg_789qwerty%2Crds%3APC_9876543210%7CPROD_PC_9876543210%2Cgpcid%3A9876543210%2Cpvt%3Ahg&hl=en&gl=us&udm=28"
}

Local Development

To run this Actor locally, make sure you have Python 3.8+ and the Apify CLI installed.

1. Install dependencies:

   $pip install -r requirements.txt
   

2. Run locally using Apify CLI:

   $apify run
   

   This will execute the code and simulate the Apify platform environment locally. The inputs are read from storage/key_value_stores/default/INPUT.json and the output datasets are written to storage/datasets/default/.

Deployment

Deploy your Actor to the Apify platform using:

$apify push

Resources

• Apify Platform documentation
• Apify SDK for Python documentation
• Apify Console