Contact

If you encounter any issues or need to exchange information, please feel free to contact us through the following link: My profile

Costco Scraper: Extract Product Data & Pricing Intelligence

Introduction

Costco is the third-largest retailer in the world and the world's largest retailer of beef, poultry, organic produce, and wine, making it a crucial data source for market research, competitive analysis, and business intelligence. Costco operates dedicated websites for multiple countries including the United Kingdom (costco.co.uk), Mexico (costco.com.mx), South Korea (costco.co.kr), and Taiwan, along with Australia and Japan, each offering region-specific products, pricing, and inventory information.

The challenge of manually tracking product information, pricing trends, and availability across multiple Costco regions is time-consuming and inefficient. Our Costco Product Search Scraper addresses this challenge by automating the extraction of comprehensive product data from category pages across all major Costco international websites, providing businesses with the competitive intelligence they need to make informed decisions.

Scraper Overview

The Costco Product Search Scraper is a sophisticated data extraction tool designed to collect detailed product information from Costco's international e-commerce platforms. Like other modern e-commerce websites, Costco relies on single-page applications (SPAs) that dynamically load content using XHR/Ajax requests, making traditional scraping methods ineffective. This scraper overcomes these technical challenges by utilizing advanced techniques to access the underlying API endpoints that deliver product data.

This tool excels in extracting comprehensive product datasets from category pages, providing over 60 data points per product including pricing information, inventory status, product specifications, and customer ratings. The scraper supports multiple international Costco domains, enabling cross-market analysis and regional pricing comparisons.

Target users include e-commerce businesses, market researchers, pricing analysts, retail consultants, and competitive intelligence professionals who need accurate, up-to-date product data from one of the world's largest retailers.

Input and Output Details

Example url 1: https://www.costco.co.uk/Computers/Laptops-MacBooks/c/cos_16.1

Example url 2: https://www.costco.com.au/Electronics/TVs-Projectors/TVs/c/cos_21.1.2

Example url 3: https://www.costco.co.jp/Home-Furnishing/Bed-Bath-Towels/Pillows-Cushions/c/cos_3.5.3

Example Screenshot of product information page:

Input Format

The scraper accepts configuration through a JSON object with several key parameters:

Scrape with URLs:

{
  "max_retries_per_url": 2, // Maximum number of retry attempts for each URL if scraping fails
  "proxy": { // Proxy configuration to avoid bot detection
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL" 
    ],
    "apifyProxyCountry": "JP" // Choose a country that matches your target data location
  },
  "max_items_per_url": 20, // Limit the number of items to scrape per URL
  "urls": [ 
    "https://www.costco.co.uk/Computers/Laptops-MacBooks/c/cos_16.1"
    // Add URLs of product list pages you want to scrape
  ],
  "ignore_url_failures": true // Continue scraping even if some URLs fail
}

The urls parameter: Add the URLs of the product list pages you want to scrape. You can paste URLs one by one, or use the Bulk edit section to add a prepared list.

The ignore_url_failures parameter: If set to true, the scraper will continue running even if some URLs fail to be scraped after the maximum number of retries is reached. This ensures that one problematic URL doesn't stop your entire scraping job.

When you provide a list of URLs for scraping, all options in the "Scrape with search filters" section will be disabled. The system will only collect data from the URLs you specified.

Scrape with Search Filters:

{
  "max_retries_per_url": 2, // Maximum number of retry attempts for each search request
  "proxy": { // Proxy configuration to avoid bot detection
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL" 
    ],
    "apifyProxyCountry": "JP" // Choose a country that matches your target data location
  },
  "max_items_per_url": 20, // Total number of items you want to scrape
  "keyword": "laptop", // Search keyword to find products
  "region": "jp", // Region to search for items
  "sort_by": "price-asc", // Sort products by specific criteria
  "page": 1 // Specify the page number to start scraping from
}

The keyword parameter: Enter the keyword to search for items (e.g., "laptop", "television", "furniture", "appliances").

The region parameter: Select the region to search for items:

• "kr" - Korea
• "jp" - Japan
• "au" - Australia
• "uk" - United Kingdom
• "is" - Iceland
• "mx" - Mexico
• "tw" - Taiwan

The sort_by parameter: Sort items by specific criteria:

• "relevance" - Relevance
• "name-asc" - Name (A-Z)
• "name-desc" - Name (Z-A)
• "price-asc" - Price (Low to High)
• "price-desc" - Price (High to Low)
• "ratings-desc" - Ratings (High to Low)
• "ratings-asc" - Ratings (Low to High)
• "sellCount-desc" - Best Selling

The page parameter: Specify the page number to start scraping from (e.g., 1, 2, 3...).

When using search filters for scraping, you need to leave the urls field empty in the "Scrape with URLs" configuration.

General Options:

The max_items_per_url parameter: Limit the number of items per URL or search filters you want to scrape. The default value is 20, providing a manageable batch size while allowing for comprehensive data collection.

The max_retries_per_url parameter: Limit the number of retries for each URL or search filters if the scrape is detected as a bot or the page fails to load. The default value is 2, providing a good balance between thoroughness and efficiency.

The proxy parameter: Proxy configuration is essential for maintaining anonymity and avoiding detection. The residential proxy option ensures that your scraping activities appear as legitimate browsing, reducing the risk of being blocked or rate-limited. You should choose a country that matches the location of the website you're scraping (e.g., JP for costco.co.jp, UK for costco.co.uk, AU for costco.com.au).

Output Format

You get the output from the costco Product Search Scraper stored in a tab. The following is an example of the Information Fields collected after running the Actor.

[ // List of product information
  {
  "add_to_cart_from_plp_categories": null,
  "addable_to_cart_from_listing_page": true,
  "addon_has_pdp": null,
  "as400_discount": false,
  "available_for_pickup": false,
  "available_for_preorder": false,
  "average_rating": 0.0,
  "average_weight": "",
  "base_price": {
    "currency_iso": "GBP",
    "formatted_price_for_set_of_two_quantities": "£9,399.98",
    "formatted_value": "£4,699.99",
    "price_type": "BUY",
    "value": 4699.99
  },
  "business_delivery": false,
  "buy_now_enabled": false,
  "code": "592685",
  "coupon_discount": {
    "discount_type": "default",
    "hide_discount_calculation": false
  },
  "decal_data": [
    {
      "key": "1",
      "value": {
        "alt_text": "\"\"",
        "position": 1,
        "url": "/mediapermalink/decal-new-2020"
      }
    }
  ],
  "department_number": "",
  "description": "<div><a href=\"/concierge-services\"><img alt=\"Extended 2 Year Warranty** on Computers, Tablets, Major Appliances/ White Goods* Cameras and Camcorders\" class=\"desktop-bnr\" src=\"/mediapermalink/concierge-2year-warranty-bnr-1400\" /> <img alt=\"Extended 2 Year Warranty** on Computers, Tablets, Major Appliances/ White Goods* Cameras and Camcorders\" class=\"mobile-bnr\" src=\"/mediapermalink/concierge-2year-warranty-bnr-480\" /></a></div>\n\n<p>&nbsp;</p>\n\n<p>The Legion 9i is a game studio in your bag, built for game developers, VFX artists, and 3D pros who need power,precision, and expandability, featuring a lightweight and ultra-durable Forged Carbon chassis. With Intel&reg; CoreTMUltra processors and GeForce RTX&nbsp; laptop graphics, it delivers real-time rendering, AI-enhanced workflows, andray tracing. The Lenovo PureSight 4K Display shifts from 4K at 240Hz to FHD at 440Hz, while Legion Coldfront Vapor technology keeps performance smooth. Legion Space unifies settings, AI-powered gaming tools, and device integration, and expandable hardware ensures the Legion 9i grows with you. For next-level creation without theneed for a 3D headset, the optional Lenovo 3D Studio enables AI-driven 3D content creation with an optional 3Ddisplay.<br />\n<br />\n<strong>Forged Carbon</strong><br />\nA pioneering material as unique as your fingerprint, this one-of-a-kind process fuses eight layers of carbon fiber for enhanced durability and reduced weight.<br />\n<br />\n<strong>Lenovo PureSight</strong><strong>Gaming Display w/ Dual</strong><strong>Mode</strong><br />\nA game studio in your bag, enjoymassive screen real estate for game development, video fx and gaming.Effortlessly shift from 4K at 240Hz to FHD at 440Hz for extreme speed.<br />\n<br />\n<strong>Legion Coldfront&nbsp;Vapor</strong><br />\nA testament to cutting-edge innovation ideal for sustained workflows, Legion Coldfront: Vapor deploys the future of cooling technology with massive vaporchambers, advanced quad turbo-charged fans and dedicated heat sinks for key components.<br />\n<br />\n<strong>Introducing LegionSpace &amp; Lenovo 3DStudio (Optional)</strong><br />\nThrough its unified portal, Legion Space provides access to custom system settings, exclusive deals, and diverse AI-powered tools, while the optional Lenovo 3D Studio software utilizes advanced AI for immersive 3D content creation and consumption on select displays.</p>\n",
  "display_ex_vat_price": false,
  "display_shipping_included_tag": false,
  "english_name": "Lenovo Legion 9, Intel Core Ultra 9, 64GB RAM, 2TB SSD, NVIDIA GeForce RTX 5090, 18 inch, Gaming Laptop, 83EY000TUK",
  "exclude_payment_plan": false,
  "expedited_delivery": false,
  "group_id": "default-cartgroup-uk",
  "has_other_options_variants": false,
  "has_price_per_unit": false,
  "hide_price_value": false,
  "id_verification_enabled": false,
  "images": [
    {
      "format": "thumbnail",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/he9/h4f/369064884404254.jpg"
    },
    {
      "format": "product",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/h9e/he6/369064883355678.jpg"
    },
    {
      "format": "results",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/h97/h5f/369064883617822.jpg"
    },
    {
      "format": "carousel",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/h97/h5f/369064883617822.jpg"
    },
    {
      "format": "thumbnail-webp",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/h5f/h9f/369064885977118.webp"
    },
    {
      "format": "product-webp",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/h43/hf8/369064886435870.webp"
    },
    {
      "format": "results-webp",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/ha2/h40/369064886894622.webp"
    },
    {
      "format": "carousel-webp",
      "image_type": "PRIMARY",
      "url": "/medias/sys_master/images/ha2/h40/369064886894622.webp"
    }
  ],
  "in_my_wishlist": false,
  "in_restricted_zone": false,
  "ineligible_to_leave_unattended": false,
  "is_base_product": false,
  "is_external_vendor_product": false,
  "is_external_vendor_variant_product": false,
  "lexical_score": 0.0,
  "max_order_quantity": 500,
  "max_order_quantity_per_date_range": 0,
  "membership": false,
  "membership_restriction_applied": false,
  "min_order_quantity": 1,
  "minimum_verification_age": 0,
  "modulus_item_quantity": 1,
  "multi_product": false,
  "name": "Lenovo Legion 9, Intel Core Ultra 9, 64GB RAM, 2TB SSD, NVIDIA GeForce RTX 5090, 18 inch, Gaming Laptop, 83EY000TUK",
  "name_in_selected_language": "Lenovo Legion 9, Intel Core Ultra 9, 64GB RAM, 2TB SSD, NVIDIA GeForce RTX 5090, 18 inch, Gaming Laptop, 83EY000TUK",
  "normalized_lexical_score": 0.0,
  "normalized_vector_score": 0.0,
  "overlay_required": false,
  "potential_promotions": [],
  "price": {
    "currency_iso": "GBP",
    "formatted_price_for_set_of_two_quantities": "£9,399.98",
    "formatted_value": "£4,699.99",
    "price_type": "BUY",
    "value": 4699.99
  },
  "price_range": {},
  "product_cart_group_available_in_warehouse": false,
  "purchasable": true,
  "random_weight": false,
  "requires_age_verification": false,
  "restricted": false,
  "rolling_time_period": 0,
  "score": 1.0,
  "shipping_included": true,
  "shipping_purchase_method_exception_message": "",
  "shipping_restriction_applied": false,
  "show_energy_rating": false,
  "show_estimated_delivery_date": false,
  "sold_individually": true,
  "stock": {
    "contact_day": 0,
    "delivery_lead_time": 0,
    "stock_level_status": "inStock"
  },
  "summary": "<p>- Intel&reg; Core&trade; Ultra 9-275HX Processor<br />-&nbsp;NVIDIA GeForce RTX&trade; 5090 graphics<br />- Windows 11 Home<br />- 16 inch WQXGA 240Hz Display<br />- Up to 4 hours Battery life<br />- 3.75kg<br />- 2 Year Warranty Included</p>",
  "tire": false,
  "url": "/Computers/Laptops-MacBooks/Lenovo-Legion-9-Intel-Core-Ultra-9-64GB-RAM-2TB-SSD-NVIDIA-GeForce-RTX-5090-18-inch-Gaming-Laptop-83EY000TUK/p/592685",
  "valid_ex_vat_data": true,
  "vector_score": 0.0,
  "volume_prices_flag": false,
  "warehouse_fulfilled": false,
  "warehouse_hide_price_value": false,
  "warehouse_pickup_only": false,
  "from_url": "https://www.costco.co.uk/Computers/Laptops-MacBooks/c/cos_16.1"
}, // ... Many other product details
]

The scraper generates a comprehensive dataset with over 60 data fields per product, providing extensive product intelligence:

Core Product Information:

• Name/Description: Product title and detailed description for identification and categorization
• Code/ID: Unique product identifiers for inventory management and cross-referencing
• Price/Base Price: Current pricing and original pricing for margin analysis
• Images: Product image URLs for visual product catalogs

Inventory and Availability:

• Stock: Current inventory status for supply chain planning
• Available for Pickup/Preorder: Fulfillment options indicating product availability
• Warehouse Fulfilled: Distribution method affecting delivery times
• In Restricted Zone: Geographic availability limitations

Business Intelligence Fields:

• Average Rating: Customer satisfaction metrics for quality assessment
• Coupon Discount/AS400 Discount: Active promotional pricing
• Membership Restrictions: Access requirements affecting target market
• Price Range: Pricing variations for different product configurations

Technical Specifications:

• Department Number: Internal categorization for organizational analysis
• Group ID: Product family relationships for bundling opportunities
• Max/Min Order Quantity: Purchase constraints affecting sales volume
• Average Weight: Shipping cost calculations

Marketing and Sales Data:

• Buy Now Enabled: E-commerce functionality status
• Expedited Delivery: Premium fulfillment options
• Potential Promotions: Upcoming marketing opportunities
• Volume Prices Flag: Bulk pricing availability

Each extracted product includes complete metadata enabling comprehensive market analysis, competitive benchmarking, and strategic business decisions.

Usage Guide

Method 1: Scrape with URLs

Step-by-Step Implementation:

1. Configuration Setup

   • Add specific product list page URLs from supported Costco domains (Korea, Japan, Australia, UK, Iceland, Mexico, Taiwan) to the urls array
   • Configure proxy settings matching the target geographic region (e.g., JP for Japan, UK for United Kingdom)
   • Set appropriate item limits based on data requirements (typically 10-50 items per URL)
   • Enable ignore_url_failures to ensure the scraper continues even if some URLs fail

2. URL Selection

   • Choose category or product listing URLs from your target Costco region
   • You can paste URLs one by one, or use the Bulk edit section to add a prepared list
   • Focus on specific categories relevant to your analysis (electronics, appliances, furniture, etc.)

3. Execution Best Practices

   • Always use residential proxies to avoid detection
   • Implement reasonable delays between requests (recommended: 1-3 seconds)
   • Start with smaller datasets to test configuration before scaling
   • Monitor for website structure changes that may affect data extraction

4. Data Processing

   • Validate extracted data for completeness and accuracy
   • Remove duplicate entries using product codes or URLs
   • Normalize pricing data across different currencies and regions
   • Structure the data in user-friendly formats such as CSV, Excel, or JSON for easier analysis

Method 2: Scrape with Search Filters

Step-by-Step Implementation:

1. Configuration Setup

   • Leave the urls field empty when using search filters
   • Configure proxy settings matching the target geographic region
   • Set appropriate item limits based on data requirements

2. Search Criteria Definition

   • Keyword: Enter the product search term (e.g., "laptop", "television", "refrigerator")
   • Region: Select your target market (Korea, Japan, Australia, UK, Iceland, Mexico, or Taiwan)
   • Sort By: Choose sorting criteria (relevance, price, ratings, or best selling)
   • Start Page: Specify which page to begin scraping from

3. Execution Best Practices

   • Always use residential proxies to avoid detection
   • Match proxy country with the selected region for optimal results
   • Start with broad keyword searches and refine based on results
   • Monitor the scraping progress through built-in logging

4. Data Processing

   • Validate extracted data for completeness and accuracy
   • Remove duplicate entries using product codes or URLs
   • Normalize pricing data across different currencies and regions
   • Structure the data in user-friendly formats such as CSV, Excel, or JSON for easier analysis

Common Error Handling:

• Blocked Requests: Adjust proxy settings or increase retry attempts
• Network Issues: Increase max_retries_per_url for temporary connection problems
• Invalid URLs: Verify URL formats match expected category page structures
• Website Changes: Check for website updates that may require scraper adjustments
• Empty Results: Ensure search keywords are appropriate for the selected region
• Incomplete Data: Implement data validation checks to identify extraction issues

Tips for Optimal Results:

• Proxy Configuration: Always match proxy location with target region (e.g., JP proxy for Japan Costco)
• Off-Peak Scraping: Schedule scraping during off-peak hours to minimize detection risk
• Start Small: Begin with limited items (10-20) to test configuration before scaling
• Regular Monitoring: Monitor data quality and extraction reliability regularly
• Filter Optimization: When using search filters, start broad and gradually refine to capture all relevant products
• URL vs Filters: Use URLs for specific categories, use filters for keyword-based product searches
• Data Validation: Implement validation checks for critical fields like pricing, availability, and product details
• Region-Specific: Understand regional differences in product availability and pricing structures

Benefits and Applications

Time and Resource Savings

This automated solution eliminates the need for manual data collection across multiple Costco websites, reducing research time from days to hours. Scraping Costco product details helps extract detailed product information, including descriptions, specifications, and pricing, which is crucial for understanding product features and pricing structures.

Real-World Applications

• Competitive Pricing Analysis: Compare prices across international markets to identify pricing opportunities
• Market Research: Analyze product availability and popularity across different regions
• Inventory Planning: Monitor stock levels and product lifecycles for supply chain optimization
• Brand Intelligence: Track competitor products and pricing strategies across global markets

Business Value

Combining Costco product data with other retail data creates a clear picture of the market at large, making it easier to discover your place within it. The comprehensive dataset enables data-driven pricing strategies, market positioning decisions, and competitive advantage development.

Regular data collection provides trending insights, seasonal pattern recognition, and market opportunity identification that would be impossible to achieve through manual research methods.

Conclusion

The Costco Product Search Scraper provides comprehensive, reliable access to one of the world's largest retailers' product databases across multiple international markets. With its extensive 60+ data fields per product and support for major Costco regions, this tool delivers the competitive intelligence needed for informed business decisions.

Ready to transform your market research capabilities? Start extracting valuable Costco product data today and gain the competitive edge your business needs.

Your feedback

We are always working to improve Actors' performance. So, if you have any technical feedback about costco Product Search Scraper or simply found a bug, please create an issue on the Actor's Issues tab in Apify Console.