Woolworths.com.au Product Search Scraper: Extract Australian Grocery Data

Understanding Woolworths.com.au and Its Market Significance

Woolworths is Australia's largest supermarket chain, serving millions of customers across 1,000+ stores nationwide. The platform offers comprehensive online shopping with 20,000+ products spanning fresh produce, groceries, household items, and specialty goods. Unlike smaller retailers, Woolworths provides real-time inventory, competitive pricing, and market-leading product range.

For price intelligence firms, the platform reveals pricing strategies across categories and regions. Retailers benchmark against Woolworths' dominant market position. Consumer goods manufacturers track their product placement, pricing, and promotional activities. Market researchers analyze product trends, seasonal availability, and consumer preferences through Woolworths' extensive catalog.

Manual collection across thousands of products and categories requires endless scrolling and data entry. This scraper automates extraction from search result pages, transforming product listings into structured datasets for analysis, monitoring, or competitive intelligence.

What This Scraper Extracts and Target Users

The Woolworths Product Search Scraper processes search result pages—the listings displayed after searching for products. It extracts product arrays with name and display name fields from each search page.

Data Fields:

Products: Array containing all product listings from the search page. Each product represents an individual item matching search criteria.

Name: Internal product identifier or SKU-based name used in Woolworths' system. Technical naming convention for backend categorization.

Display Name: Customer-facing product name shown on website. Marketing-optimized description including brand, size, and key attributes (e.g., "Woolworths Beef Burgers 500g").

Target Audiences:

Price Intelligence Companies monitor pricing across grocery categories and competitors. Retail Competitors benchmark product ranges, naming conventions, and category organization. CPG Manufacturers track their brand presence, product positioning, and market share indicators. Market Researchers analyze product availability, trending items, and seasonal patterns. E-commerce Platforms build product databases or comparison engines using structured grocery data.

Input Configuration: Search URL Structure

The scraper processes Woolworths search result URLs containing product listings, not individual product detail pages.

Example Input:

{
  "proxy": {
    "useApifyProxy": false
  },
  "max_items_per_url": 20,
  "ignore_url_failures": true,
  "urls": [
    "https://www.woolworths.com.au/shop/search/products?searchTerm=Burgers&pageNumber=2"
  ]
}

Example Screenshot:

Parameter Breakdown:

proxy.useApifyProxy: Set false if using custom proxies or direct access. Enable (true) for Apify's proxy network when facing access restrictions. Australian proxies recommended for regional accuracy.

max_items_per_url: Limits products extracted per search page. Default 20 matches typical search page display. Increase to 50-100 for comprehensive extraction when pages contain more results.

ignore_url_failures: Enables continued processing if URLs fail. Critical for bulk scraping—one invalid URL won't halt entire job. Set false only when every URL must succeed.

urls array: Contains Woolworths search result URLs. Format: https://www.woolworths.com.au/shop/search/products?searchTerm=[KEYWORD]&pageNumber=[N]. Collect URLs by performing searches on Woolworths, then incrementing pageNumber for pagination.

Building URL Lists: Search for categories (dairy, meat, bakery) or specific terms (organic, gluten-free) manually on Woolworths. Copy resulting URLs. For multi-page results, increment pageNumber parameter to capture complete datasets.

Complete Output Structure and Field Definitions

Products Array: Contains all product objects from the search page. Each object represents one product listing with name fields.

Name Field: Technical product identifier, typically SKU-based or internal naming convention. Format varies but generally includes category codes or system identifiers. Used for backend tracking, inventory management, and technical integrations. Less human-readable but consistent for database operations.

Display Name Field: Customer-facing product name optimized for shoppers. Includes brand, product type, size/weight, and key attributes. Examples: "Coles Bakery Beef Burgers 8 Pack", "Macro Organic Grass Fed Beef Burgers 400g". This field powers customer search, filtering, and product discovery. Marketing teams analyze naming conventions, while researchers track product positioning.

Sample Output:

[
  {
  "products": [
    {
      "tile_i_d": 1,
      "stockcode": 901682,
      "barcode": "9339687384471",
      "gtin_format": 13,
      "cup_price": 11.82,
      "instore_cup_price": 11.82,
      "cup_measure": "1KG",
      "cup_string": "$11.82 / 1KG",
      "instore_cup_string": "$11.82 / 1KG",
      "has_cup_price": true,
      "instore_has_cup_price": true,
      "price": 6.5,
      "incremental_price": null,
      "incremental_was_price": null,
      "incremental_minimum_quantity": null,
      "incremental_minimum_quantity_unit": "g",
      "price_per_k_g_label": "Price per Kg charged",
      "min_weight_price": null,
      "max_weight_price": null,
      "instore_price": 6.5,
      "name": "Woolworths 10 Chicken Sausages",
      "display_name": "Woolworths 10 Chicken Sausages 550g",
      "url_friendly_name": "woolworths-10-chicken-sausages",
      "description": " Woolworths 10 Chicken<br>Sausages  550g",
      "small_image_file": "https://cdn1.woolworths.media/content/wowproductimages/small/901682.jpg",
      "medium_image_file": "https://cdn1.woolworths.media/content/wowproductimages/medium/901682.jpg",
      "large_image_file": "https://cdn1.woolworths.media/content/wowproductimages/large/901682.jpg",
      "is_new": false,
      "is_half_price": false,
      "is_online_only": false,
      "is_on_special": false,
      "instore_is_on_special": false,
      "is_edr_special": false,
      "savings_amount": 0.0,
      "instore_savings_amount": 0.0,
      "was_price": 6.5,
      "instore_was_price": 6.5,
      "quantity_in_trolley": 0,
      "unit": "Each",
      "minimum_quantity": 1,
      "has_been_bought_before": false,
      "is_in_trolley": false,
      "source": "SearchServiceSearchProducts",
      "supply_limit": 20,
      "product_limit": 36,
      "max_supply_limit_message": "20 item(s) left",
      "is_ranged": true,
      "is_in_stock": true,
      "package_size": "550g",
      "is_pm_delivery": false,
      "is_for_collection": true,
      "is_for_delivery": true,
      "is_for_express": true,
      "product_restriction_message": null,
      "product_warning_message": null,
      "centre_tag": {
        "tag_content": "<a title=\"2 for $12.00\" href=\"/Shop/OfferGroup/22349311\"><span>2 for $12.00 - $10.91/1KG</span><i aria-hidden=\"true\" class=\"iconAct-Chevron_Right\"></i></a> ",
        "tag_link": "",
        "fallback_text": null,
        "tag_type": "Html",
        "multibuy_data": {
          "quantity": 2,
          "price": 12,
          "cup_tag": "$10.91/1KG"
        },
        "member_price_data": null,
        "f_f_p_v_member_price_data": null,
        "tag_content_text": "2 for $12.00 - $10.91/1KG ",
        "dual_image_tag_content": null,
        "promotion_type": "NOT_SET",
        "is_registered_reward_card_promotion": false
      },
      "is_centre_tag": true,
      "image_tag": {
        "tag_content": "/content/promotiontags/roundel-buy-more-save-more-200x200.png",
        "tag_link": null,
        "fallback_text": "Buy More Save More",
        "tag_type": "Image",
        "multibuy_data": null,
        "member_price_data": null,
        "f_f_p_v_member_price_data": null,
        "tag_content_text": null,
        "dual_image_tag_content": null,
        "promotion_type": "NOT_SET",
        "is_registered_reward_card_promotion": false
      },
      "header_tag": null,
      "has_header_tag": false,
      "unit_weight_in_grams": 0,
      "supply_limit_message": "'Woolworths 10 Chicken Sausages' has a supply limit of 20. The quantity in your cart has been reduced accordingly. To purchase a larger quantity, please contact us on 1800 000 610. Please note we do not supply trade orders.",
      "small_format_description": "Woolworths 10 Chicken Sausages ",
      "full_description": "Woolworths 10 Chicken Sausages ",
      "is_available": true,
      "instore_is_available": true,
      "is_purchasable": true,
      "instore_is_purchasable": true,
      "age_restricted": false,
      "display_quantity": 1,
      "rich_description": null,
      "hide_was_saved_price": false,
      "sap_categories": null,
      "brand": "Woolworths",
      "is_restricted_by_delivery_method": false,
      "footer_tag": {
        "tag_content": null,
        "tag_link": null,
        "fallback_text": null,
        "tag_type": "None",
        "multibuy_data": null,
        "member_price_data": null,
        "f_f_p_v_member_price_data": null,
        "tag_content_text": null,
        "dual_image_tag_content": null,
        "promotion_type": "NOT_SET",
        "is_registered_reward_card_promotion": false
      },
      "is_footer_enabled": false,
      "diagnostics": "48",
      "is_bundle": false,
      "is_in_family": false,
      "child_products": null,
      "url_override": null,
      "additional_attributes": {
        "boxedcontents": null,
        "addedvitaminsandminerals": "False",
        "sapdepartmentname": "FRESH CONVENIENCE",
        "spf": null,
        "haircolour": null,
        "lifestyleanddietarystatement": "Gluten Free,Low Sugar",
        "sapcategoryname": "MEAT CONVENIENCE",
        "skintype": null,
        "importantinformation": null,
        "allergystatement": "Dairy Free,Egg Free,Fish Free,Gluten Free,Lactose Free,No Artificial Colours or Flavours,Soy Free,Wheat Free",
        "productdepthmm": null,
        "skincondition": null,
        "ophthalmologistapproved": null,
        "healthstarrating": "3.5",
        "hairtype": null,
        "fragrance-free": null,
        "sapsegmentname": "CORE SAUSAGES",
        "suitablefor": null,
        "pies_product_departmentsjson": "[{\"Id\":\"1_8AD6702\",\"Description\":\"Dinner\"},{\"Id\":\"1_D5A2236\",\"Description\":\"Poultry, Meat & Seafood\"}]",
        "piessubcategorynamesjson": "[\"Sausages & Kebabs\",\"Burgers & Sausages\",\"Sausages\"]",
        "sapsegmentno": "7",
        "productwidthmm": null,
        "contains": null,
        "sapsubcategoryname": "SAUSAGES",
        "dermatologisttested": null,
        "wool_productpackaging": null,
        "dermatologicallyapproved": null,
        "specialsgroupid": null,
        "productimages": "901682.jpg,901682_2.jpg",
        "productheightmm": null,
        "r&r_hidereviews": null,
        "microwavesafe": "False",
        "paba-free": null,
        "lifestyleclaim": null,
        "alcoholfree": null,
        "tgawarning": null,
        "activeconstituents": null,
        "microwaveable": "False",
        "soap-free": null,
        "countryoforigin": null,
        "isexcludedfromsubstitution": "False",
        "productimagecount": "2",
        "r&r_loggedinreviews": null,
        "anti-dandruff": null,
        "servingsize-total-nip": null,
        "tgahealthwarninglink": null,
        "allergenmaybepresent": null,
        "pies_product_department_node_id": "1_8AD6702",
        "parabenfree": "False",
        "vendorarticleid": null,
        "containsgluten": "False",
        "containsnuts": "False",
        "ingredients": "RSPCA Approved Chicken (76%), Water, Seasoning (Potato Starch, Rice Flour, Maize Starch, Salt, Plant Fibres (Bamboo, Pea, Psyllium Husk), Vegetable Powders (Onion, Garlic), Mineral Salt (451), Yeast Extract, Preservative (223 (Sulphites)), Spices, Natural Flavour, Herbs), Vegetable Casing (Water, Thickener (401), Stabiliser (Guar Gum), Acidity Regulators (Citric Acid, Lactic Acid), Preservative (223 (Sulphites)), Firming Agent (509))",
        "colour": null,
        "manufacturer": null,
        "sapcategoryno": "1",
        "storageinstructions": "Keep Refrigerated",
        "tgawarnings": null,
        "piesdepartmentnamesjson": "[\"Dinner\",\"Poultry, Meat & Seafood\"]",
        "brand": "woolworths",
        "oilfree": null,
        "fragrance": null,
        "antibacterial": "False",
        "non-comedogenic": null,
        "antiseptic": "False",
        "bpafree": "False",
        "vendorcostprice": null,
        "description": null,
        "sweatresistant": null,
        "sapsubcategoryno": "327",
        "antioxidant": "False",
        "claims": null,
        "phbalanced": null,
        "wool_dietaryclaim": null,
        "ophthalmologisttested": null,
        "sulfatefree": "False",
        "piescategorynamesjson": "[\"BBQ Ready\",\"Meat\",\"BBQ Meat\"]",
        "servingsperpack-total-nip": null,
        "nutritionalinformation": "{\"Name\":\"Nutritional Information\",\"Attributes\":[{\"Id\":705,\"Name\":\"Carbohydrate Quantity Per 100g - Total - NIP\",\"Value\":\"5.3g\",\"Description\":\"Carbohydrate Quantity Per 100g - Total - NIP\",\"SortOrder\":83},{\"Id\":759,\"Name\":\"Energy kJ Quantity Per 100g - Total - NIP\",\"Value\":\"628.0kJ\",\"Description\":\"Energy kJ Quantity Per 100g - Total - NIP\",\"SortOrder\":239},{\"Id\":253,\"Name\":\"Fat Saturated Quantity Per 100g - Total - NIP\",\"Value\":\"2.4g\",\"Description\":\"Fat Saturated Quantity Per 100g - Total - NIP\",\"SortOrder\":251},{\"Id\":764,\"Name\":\"Fat Total Quantity Per 100g - Total - NIP\",\"Value\":\"8.0g\",\"Description\":\"Fat Total Quantity Per 100g - Total - NIP\",\"SortOrder\":263},{\"Id\":166,\"Name\":\"Fat Total Quantity Per 100g - ValueWord - NIP\",\"Value\":\"8.0\",\"Description\":\"Fat Total Quantity Per 100g - ValueWord - NIP\",\"SortOrder\":264},{\"Id\":878,\"Name\":\"Protein Quantity Per 100g - Total - NIP\",\"Value\":\"14.2g\",\"Description\":\"Protein Quantity Per 100g - Total - NIP\",\"SortOrder\":587},{\"Id\":477,\"Name\":\"Serving Size - Total - NIP\",\"Value\":\"55.0 G\",\"Description\":\"Serving Size - Total - NIP\",\"SortOrder\":607},{\"Id\":544,\"Name\":\"Servings Per Pack - Total - NIP\",\"Value\":\"10.0\",\"Description\":\"Servings Per Pack - Total - NIP\",\"SortOrder\":611},{\"Id\":491,\"Name\":\"Sodium Quantity Per 100g - Total - NIP\",\"Value\":\"502.0mg\",\"Description\":\"Sodium Quantity Per 100g - Total - NIP\",\"SortOrder\":623},{\"Id\":909,\"Name\":\"Sugars Quantity Per 100g - Total - NIP\",\"Value\":\"1.0g\",\"Description\":\"Sugars Quantity Per 100g - Total - NIP\",\"SortOrder\":647},{\"Id\":910,\"Name\":\"Sugars Quantity Per 100g - ValueWord - NIP\",\"Value\":\"1.0\",\"Description\":\"Sugars Quantity Per 100g - ValueWord - NIP\",\"SortOrder\":648}]}",
        "ovencook": "False",
        "vegetarian": "False",
        "hypo-allergenic": null,
        "timer": null,
        "dermatologistrecommended": null,
        "sapdepartmentno": "10",
        "allergencontains": "Sulphites",
        "waterresistant": null,
        "friendlydisclaimer": null,
        "recyclableinformation": null,
        "usageinstructions": null,
        "freezable": "False"
      },
      "details_image_paths": [
        "https://cdn1.woolworths.media/content/wowproductimages/large/901682.jpg"
      ],
      "variety": null,
      "rating": {
        "review_count": 0,
        "rating_count": 0,
        "rating_sum": 0,
        "one_star_count": 0,
        "two_star_count": 0,
        "three_star_count": 0,
        "four_star_count": 0,
        "five_star_count": 0,
        "average": 0,
        "one_star_percentage": 0,
        "two_star_percentage": 0,
        "three_star_percentage": 0,
        "four_star_percentage": 0,
        "five_star_percentage": 0
      },
      "has_product_subs": false,
      "is_sponsored_ad": false,
      "ad_i_d": null,
      "ad_index": null,
      "ad_status": null,
      "is_market_product": false,
      "is_giftable": false,
      "vendor": null,
      "untraceable": false,
      "third_party_product_info": null,
      "market_features": null,
      "market_specifications": null,
      "supply_limit_source": "StockQuantity",
      "tags": [
        {
          "content": {
            "type": "Roundel",
            "position": "Top",
            "attributes": {
              "image_path": "/content/promotiontags/roundel-buy-more-save-more-200x200.png",
              "fallback_text": "Buy More Save More"
            },
            "f_f_p_v_attributes": null
          },
          "template_id": null,
          "metadata": null
        },
        {
          "content": {
            "type": "PromotionInformation",
            "position": "Top",
            "attributes": {
              "tag_link": "",
              "text": "2 for $12.00 - $10.91/1KG ",
              "html": "<a title=\"2 for $12.00\" href=\"/Shop/OfferGroup/22349311\"><span>2 for $12.00 - $10.91/1KG</span><i aria-hidden=\"true\" class=\"iconAct-Chevron_Right\"></i></a> ",
              "is_registered_reward_card_promotion": "False"
            },
            "f_f_p_v_attributes": null
          },
          "template_id": null,
          "metadata": null
        }
      ],
      "is_personalised_by_purchase_history": false,
      "is_from_faceted_search": false,
      "next_availability_date": "2026-01-05T00:00:00.0000000Z",
      "number_of_substitutes": 0,
      "is_primary_variant": false,
      "variant_group_id": 0,
      "has_variants": false,
      "variant_title": null,
      "is_tobacco": false,
      "is_free_shipping": null,
      "fulfilment_store_id": 1101,
      "b2_b_extended_range": null,
      "offer_id": null,
      "bundle_product_groups": null
    }
  ],
  "name": "Woolworths 10 Chicken Sausages",
  "display_name": "Woolworths 10 Chicken Sausages 550g",
  "from_url": "https://www.woolworths.com.au/shop/search/products?searchTerm=Burgers&pageNumber=2"
}
]

Data Applications:

Name field enables database primary keys, tracking products across sessions, and linking to external inventory systems. Display Name powers customer-facing applications, competitive analysis of product positioning, and natural language processing for trend identification.

Step-by-Step Implementation Guide

1. Define Target Products: Identify categories or keywords needed. Grocery staples (milk, bread), dietary segments (vegan, keto), or specific brands. Perform test searches on Woolworths to verify results match requirements.

2. Collect Search URLs: Execute searches on Woolworths.com.au. Copy URLs from browser. For comprehensive category extraction, note total page count and create URL list with incremented pageNumber values.

3. Configure Input JSON: Populate urls array with collected search URLs. Set max_items_per_url based on expected results per page (20-30 typical, adjust higher for dense categories). Enable ignore_url_failures for robustness.

4. Execute Scraping Run: Launch via Apify console. Monitor real-time progress. Processing 10 search pages typically completes in 2-4 minutes depending on network and proxy performance.

5. Review Output Quality: Check dataset preview. Verify products array populated with name and display_name. Confirm data matches expected search terms and categories.

6. Export Data: Download JSON for databases, CSV for spreadsheet analysis, or Excel for business reporting. Clean data by removing duplicates if same product appears across multiple searches.

7. Handle Pagination Systematically: For large categories, calculate total pages needed. Generate URL lists programmatically by incrementing pageNumber. Example: Burgers category with 100 products across 5 pages requires URLs with pageNumber=1 through pageNumber=5.

Error Prevention: Verify URLs are search result pages, not product details or category pages. Ensure searchTerm parameter contains valid values. Invalid pageNumber values (beyond available pages) may return empty results—check total pages before building URL lists.

Strategic Applications for Retail Intelligence

Price Monitoring Foundations: While this scraper extracts names, combine with product detail scrapers to build comprehensive pricing databases. Track which products Woolworths stocks, establishing baseline for price tracking systems.

Competitive Product Range Analysis: Compare product counts across categories versus competitors. Identify gaps—categories where Woolworths has limited offerings versus extensive ranges. Reveals strategic focus areas and underserved segments.

Brand Presence Tracking: Extract all products containing specific brand names. Monitor brand shelf space indicators—product count, naming prominence, and category representation. CPG manufacturers assess retail presence and competitive positioning.

Product Naming Intelligence: Analyze display_name conventions across categories. Identify naming trends—emphasis on organic, size descriptions, premium positioning terms. Optimize your own product naming based on market leader conventions.

Category Depth Measurement: Count unique products per category search. Track expansion over time—growing product counts signal category investment. Declining counts may indicate rationalization or shifting strategies.

Seasonal Availability Tracking: Scrape same searches monthly to identify seasonal products appearing/disappearing. Forecast inventory needs based on Woolworths' seasonal patterns—holiday items, summer produce, winter comfort foods.

New Product Detection: Regular scraping identifies new products through name field changes. Early detection of competitor launches or category innovations. Alert systems trigger when new display names appear in tracked searches.

Market Segmentation Analysis: Search dietary-specific terms (gluten-free, vegan, keto). Product count and naming conventions reveal market size and Woolworths' segmentation strategy. Manufacturers assess category viability before product development.

Maximizing Data Value: Advanced Techniques

Build Historical Databases: Store scraped data with timestamps. Track product additions/removals over time. Historical name data reveals product lifecycle—launches, rebrands, discontinuations. Multi-year datasets show category evolution.

Combine with Product Detail Scrapers: Use search scraper to collect product names/URLs, then feed to detail scrapers for pricing, specifications, and images. Two-stage approach efficiently builds comprehensive product databases.

Cross-Reference with Competitors: Scrape equivalent searches on Coles, IGA, or specialty retailers. Compare product availability, naming conventions, and category depth. Identify exclusive products or gaps in competitor offerings.

Natural Language Processing: Analyze display_name text for keyword frequencies. Extract brand mentions, size patterns, and descriptor terms (organic, premium, value). Quantify marketing positioning across thousands of products.

Geographic Availability Mapping: If Woolworths shows regional product variations, scrape from different Australian proxy locations. Identify products exclusive to certain states or metro areas. Regional retailers adjust strategies accordingly.

Category Taxonomy Development: Use search results to build product classification systems. Group similar display names into subcategories. Develop standardized taxonomy for cross-retailer comparisons or internal categorization.

Promotional Activity Indicators: While basic scraper captures names, changes in product order or appearance in search results may indicate promotions. Products moving to top positions in subsequent scrapes signal promotional boosts.

Data Quality and Operational Best Practices

Scraping Frequency: Grocery inventory changes frequently. Weekly scraping captures new products and discontinuations. Daily scraping for high-priority categories or during promotional periods. Balance data freshness against resource usage.

Deduplication Strategy: Same product may appear in multiple search results. Use name field as unique identifier to remove duplicates when combining datasets from different searches.

URL Validation: Test URLs in browser before bulk scraping. Woolworths may update search parameter names or URL structure. Invalid URLs waste resources and skew results.

Data Storage Organization: Store results with search term, page number, and scrape timestamp. Enables tracking which searches yielded which products and when. Critical for change detection and historical analysis.

Error Handling: Log failed URLs separately. Common failures include invalid page numbers (requesting page 10 when only 5 exist) or temporarily unavailable search functionality. Retry logic improves data completeness.

Rate Limiting Respect: Space out large scraping jobs. Processing 100+ URLs consecutively may trigger rate limits. Implement delays between requests or distribute across multiple runs. Sustainable practices ensure long-term access.

Enrichment Pipelines: Raw name/display_name data forms foundation for enrichment. Match products to external databases using name fields. Append pricing from detail scrapers, nutritional data from manufacturer sites, or reviews from consumer platforms.

Conclusion

The Woolworths.com.au Product Search Scraper provides foundational product intelligence from Australia's retail leader. While focused on name extraction, this data enables product tracking, competitive analysis, and market research across 20,000+ grocery items. Whether monitoring brand presence, analyzing category depth, or building pricing databases, structured product names drive retail intelligence. Start extracting Australian grocery insights today.