LIDL Product Search Scraper: Extract Retail Data at Scale

---

Why Scrape LIDL?

LIDL operates across Europe and beyond, offering thousands of products through regional websites (lidl.co.uk, lidl.de, lidl.fr, etc.). For price analysts, retail researchers, and e-commerce professionals, manually collecting product data from LIDL search results is inefficient. The LIDL Product Search Scraper automates this process, extracting comprehensive product information from search results and category pages in seconds.

---

What Does This Scraper Do?

The LIDL Product Search Scraper targets LIDL search result pages and category listings, collecting detailed product records with 54+ fields per item. It is ideal for:

• Price intelligence teams monitoring competitor pricing and promotions
• Retail researchers analyzing product assortments by region
• E-commerce aggregators building product feeds from LIDL
• Market analysts tracking promotional campaigns (flash sales, LIDL Plus deals)
• Supply chain professionals monitoring stock availability across regions

The scraper handles multiple LIDL domains seamlessly, supports pagination through offset parameters, and includes error resilience via the ignore_url_failures flag.

---

Input Configuration

The scraper accepts a simple JSON configuration:

{
  "urls": [
    "https://www.lidl.co.uk/c/food-drink/s10068374?offset=24"
  ],
  "ignore_url_failures": true,
  "max_items_per_url": 200
}

Input Fields Explained

Parameter	Type	Description
urls	array	Direct links to LIDL search results or category pages. Supports any LIDL domain (lidl.co.uk, lidl.de, lidl.fr, etc.). Include offset parameters for pagination.
max_items_per_url	integer	Maximum products to collect per URL (default: 20, max recommended: 200). Higher values extract full result sets.
ignore_url_failures	boolean	If true, continues scraping even if some URLs fail. Useful for bulk runs; if false, stops on first error.

Pro Tip: Use pagination (offset=24, offset=48, etc.) to collect products across multiple pages of search results.

---

Output Data Structure

Sample output

{
  "age_restriction": false,
  "awards": [],
  "brand": {
    "name": "PARKSIDE",
    "url": "/q/search?q=parkside",
    "show_brand": true
  },
  "canonical_path": "/p/parkside-hand-rake/p10050668",
  "canonical_url": "/p/parkside-hand-rake/p10050668",
  "category": "NonFood",
  "cutoutimage_v2": [],
  "deal_of_day": {
    "active": false
  },
  "design_theme": "default",
  "disclaimers": [
    {
      "id": "code_10050668_9b98da5e",
      "type": "CODE",
      "value": "Basic"
    }
  ],
  "display_free_delivery": false,
  "energy_labels": [],
  "erp_number": "10050668",
  "flash_sales": false,
  "full_title": "PARKSIDE Hand Rake",
  "gs1_attributes": [],
  "having_price": true,
  "having_threesixty": false,
  "having_videos": false,
  "highlighted_seals": 0,
  "ians": [
    "496480"
  ],
  "image": "https://imgproxy-retcat.assets.schwarz/c88Of-Rwq0fi4CVOyB5mO4qzI9vYUnafKI2fOb9fCk0/sm:1/exar:1:ce/w:427/h:320/cz/M6Ly9wcm9kLWNhd/GFsb2ctbWVkaWEvdWsvMS9GQUU1OEE3NzM5M0NCQ0U2MjdCRERENTI/5M0Q3QzdFMDBFQzEzMUIzNzY2M0Q4M0I1ODhBNjJGNkVBNUUxNjFBLmpwZw.jpg",
  "image_v1": null,
  "is_lidl_gift_card": false,
  "is_used_product": false,
  "item_id": 10050668,
  "image_list": [
    "https://imgproxy-retcat.assets.schwarz/c88Of-Rwq0fi4CVOyB5mO4qzI9vYUnafKI2fOb9fCk0/sm:1/exar:1:ce/w:427/h:320/cz/M6Ly9wcm9kLWNhd/GFsb2ctbWVkaWEvdWsvMS9GQUU1OEE3NzM5M0NCQ0U2MjdCRERENTI/5M0Q3QzdFMDBFQzEzMUIzNzY2M0Q4M0I1ODhBNjJGNkVBNUUxNjFBLmpwZw.jpg"
  ],
  "image_list_v1": null,
  "keyfacts": {
    "analytics_category": "NonFood",
    "description": "<ul> <li>Made from stainless steel</li> <li>Handle made of beech wood</li> <li>For raking, cleaning and loosening as well as smoothing the soil</li> <li>With a practical PU leather hanging loop</li> <li>3-year warranty</li> </ul>",
    "full_title": "PARKSIDE Hand Rake",
    "more_details": "More details",
    "title": "Hand Rake",
    "won_category_primary": "Worlds of need/DIY store & garden/Garden equipment & garden tools",
    "won_category_primary_path": "0/12/1212"
  },
  "lidl_plus": [],
  "multipack": false,
  "online": false,
  "prevent_selling": false,
  "price": {
    "base_price": {
      "prefix": false,
      "text": "each"
    },
    "currency_code": "GBP",
    "currency_code_second": "",
    "currency_symbol": "£",
    "currency_symbol_second": "",
    "displayed_currency": "£",
    "has_star": false,
    "has_vat": false,
    "old_price": 0,
    "price": 1.99,
    "price_theme": "white_red",
    "special_taxes": [],
    "variants_have_different_prices": false
  },
  "product_id": 10050668,
  "product_type": "RETAIL",
  "product_origin": "progress_event",
  "quick_add_to_cart": false,
  "regions_prices": {},
  "ribbons": [],
  "rendered_ts": 1782947580,
  "seals": [],
  "stock_availability": {
    "availability_indicator": 0,
    "badge_info": {
      "badges": [
        {
          "text": "In store from 09.07",
          "type": "IN_STORE_FROM_FUTURE_DATE_RANGE"
        }
      ]
    },
    "badge_info_v2": [
      {
        "badges": [
          {
            "text": "In store from 09.07",
            "type": "IN_STORE_FROM_FUTURE_DATE_RANGE"
          }
        ],
        "valid_from": 1783551600,
        "valid_until": 1784156399
      }
    ],
    "online_available": false,
    "orderable_quantities": [],
    "min_orderable_quantity": 1
  },
  "store": false,
  "title": "Hand Rake",
  "timezone_id": "Europe/London",
  "zones": {},
  "relevancy_score": "96129KqLblxO6sNCx6zqHiwICV153FGy0Js1IRgOc+gEG9Dqn8rcqPo8ckI=",
  "keyword_score": "lM+elch3+2jigyWJsAoGq0+FDKaaQXD5s5yPib7X0YUOHEtz9T2tyUBTYY4=",
  "semantic_score": "+2IhYu8nIRxLNE7rzpTTO+jxb/2NbzU6a+eUIzLcvXVlJ3jPsk1b9TSwS/w=",
  "keyword_score_raw": "sW1eQ36r38zRG8KCiiKK26PxL+sb+JiNK71n+v/5UK7IYdTaBdjGvgXEnsk=",
  "rating_score": "xTF59gupNe5c1a3G7GM3mIojlxu3d/6vU365X2FkoHUzmNaDcPzir7UwLg8=",
  "freshness_score": "hL2w9O7YoZJHbgGqE3wBFYKE0En9w6HjuX/ZocDfe6ahKotjSdOua/1JQFo=",
  "search_hub_impressions": "ATnH2ofuGmTW0ITFPVyRkIg6k7F0qekuq796NtmuO4GfPQzHsfc4TUdzj4s=",
  "search_hub_clicks": "WPSJUXkoZbB5gVc03nVcAjW4ZxlKEoTSAgOEvbU77mxzvZVRIKXrxU7k7n4=",
  "search_hub_carts": "kcAZC96pdEmaGQRMSc/goVvRhqj/88Zu1suEkkBx0aAQELidMZUkF0Dh4Hg=",
  "business_rule_factor": "2UnEpqdZS7UWkDbRp1HsNGyL4FsQn0nNbVNOupkrJN9+IV+fXNbqnOBVL7Y=",
  "from_url": "https://www.lidl.co.uk/q/search?q=land"
}

Each product record contains 54+ fields organized into logical categories:

Product Identity

Field	Meaning
Product ID	Unique LIDL product identifier
Item ID	Alternative internal item code
ERP Number	Enterprise Resource Planning system number
Title	Product name as displayed
Full Title	Complete, untruncated product title
Brand	Manufacturer or brand name
Product Type	Category classification (e.g., "Beverage", "Snack")
Product Origin	Country or region of manufacture

Pricing & Promotions

Field	Meaning
Price	Current retail price in local currency
Regions Prices	Price variations across different LIDL regions
Having Price	Flag indicating if price data is available
Deal Of Day	Flag if product is featured as "Deal of the Day"
Flash Sales	Active flash sale or limited-time offer
Lidl Plus	Premium pricing or discount for LIDL Plus members
Multipack	Information if product is sold as a multipack bundle

Media & Visuals

Field	Meaning
Image	Primary product image URL (standard resolution)
Image V1	Alternative image version 1
Image List	Array of all product images
Image List V1	Alternative image list version
Cutoutimage V2	High-quality cutout image (isolated product on white)
Having Videos	Flag if product has demo or tutorial videos
Having Threesixty	Flag if 360° product view is available

Product Details & Compliance

Field	Meaning
Keyfacts	Key product specifications (nutrition, ingredients, dimensions)
GS1 Attributes	Global standard product attributes and classifications
Energy Labels	EU energy efficiency labels (if applicable)
Disclaimers	Legal notices (allergen warnings, age restrictions)
Age Restriction	Minimum age requirement (e.g., alcohol products)
Is Used Product	Flag if item is refurbished or pre-owned
Is Lidl Gift Card	Flag if product is a LIDL gift card
IANs	International Article Numbers
Awards	Product certifications or awards

Availability & Store Info

Field	Meaning
Stock Availability	Current stock status (in stock, low, out of stock)
Online	Whether product is available for online purchase
Store	Physical store availability information
Prevent Selling	Flag if product cannot currently be purchased
Display Free Delivery	Whether free shipping/delivery applies
Quick Add To Cart	Flag enabling express checkout option

Visual Design & Presentation

Field	Meaning
Design Theme	Visual theme applied to product listing
Ribbons	Marketing ribbons (e.g., "New", "Organic")
Highlighted Seals	Special badges (certifications, eco-friendly)
Seals	All product seals and certifications
Canonical Path	SEO canonical URL path for the product
Canonical URL	Full canonical URL for deduplication

Search & Ranking Metrics

Field	Meaning
Relevancy Score	How relevant the product is to the search query (0-100)
Keyword Score	Match strength between product and search keywords
Keyword Score Raw	Unprocessed keyword matching score
Semantic Score	NLP-based semantic similarity to search terms
Rating Score	Customer rating influence on ranking (if available)
Freshness Score	How recent the product data is
Business Rule Factor	Internal LIDL ranking algorithm adjustment
Search Hub Impressions	Number of times product appeared in search results
Search Hub Clicks	Click-through count from search to product
Search Hub Carts	How many times users added product to cart from search

Technical Metadata

Field	Meaning
Rendered TS	Timestamp when product page was rendered/captured
Timezone ID	Timezone of the LIDL store region
Zones	Geographic zones where product is available
Category	Product category hierarchy

---

How to Use This Scraper

Step 1: Find Your URLs

Navigate to LIDL.com (or any regional LIDL domain). Use the search bar or browse categories. Copy the result page URL:

https://www.lidl.co.uk/c/food-drink/s10068374?offset=24

Step 2: Configure Input

Paste URLs into the urls array. Use multiple URLs to cover different searches or pages:

{
  "urls": [
    "https://www.lidl.co.uk/c/food-drink/s10068374?offset=0",
    "https://www.lidl.co.uk/c/food-drink/s10068374?offset=24",
    "https://www.lidl.co.uk/c/food-drink/s10068374?offset=48"
  ],
  "ignore_url_failures": true,
  "max_items_per_url": 200
}

Step 3: Run the Scraper

Start the actor and monitor logs in real-time. The scraper will collect all 54+ fields per product.

Step 4: Export Results

Download data as JSON, CSV, or Excel. Directly import into your BI tool, database, or analysis software.

Troubleshooting:

• No data returned: Verify the URL is a valid LIDL search results or category page (not a single product page)
• Incomplete data: Set max_items_per_url higher; LIDL may limit initial results
• Regional pricing differences: Use country-specific domains for accurate pricing

---

Real-World Applications

Price Monitoring: Track LIDL product prices weekly to detect promotional cycles and competitive positioning.

Promotional Analysis: Extract flash sales, LIDL Plus deals, and "Deal of Day" flags to study marketing strategies.

Assortment Research: Compare product availability and categories across LIDL regions (UK vs. Germany vs. France) to understand localization.

E-commerce Data: Aggregate LIDL products into your own retail search engine or price comparison platform.

Supply Chain: Monitor stock availability (Stock Availability field) to understand demand patterns and procurement timing.

By automating data extraction, this scraper saves 50+ hours per month compared to manual collection, enabling data-driven retail decisions.

---

Conclusion

The LIDL Product Search Scraper delivers rich, structured product data from any LIDL region in minutes. With 54+ fields covering pricing, availability, media, and search metrics, it is your complete solution for retail intelligence, price monitoring, and competitive analysis. Start scraping today and unlock insights your competitors may miss.