Lidl Category Scraper

The Lidl Category Scraper is a powerful Apify Actor designed to extract detailed product information from Lidl's online catalog based on search keywords. It leverages PlaywrightCrawler for reliable, headless browsing to gather comprehensive data on products, including prices, images, and availability, making it ideal for e-commerce analysis and monitoring. This tool ensures high-quality, structured data output to streamline your data collection workflows.

Features

• Keyword-Based Search: Quickly scrape products by entering specific search terms, allowing targeted data extraction from Lidl's vast catalog.
• Comprehensive Product Data: Retrieves key details like product ID, title, price, discounts, images, brand, category, and availability status for in-depth insights.
• Proxy Support: Utilizes residential proxies to avoid IP blocks and ensure uninterrupted scraping, enhancing reliability for large-scale operations.
• High Performance: Built on PlaywrightCrawler for fast, efficient crawling with minimal errors, supporting both online and in-store product availability.
• Structured Output: Delivers clean JSON data with fields for analytics, including relevancy scores, impressions, and click metrics for advanced use cases.
• Error Handling: Includes robust mechanisms to handle rate limits, timeouts, and dynamic content, ensuring consistent data retrieval.
• Scalable and Customizable: Easily configurable for different search queries, making it suitable for batch processing and integration with other tools.

Input Parameters

Parameter	Type	Required	Description	Example
searchKeyword	string	Yes	The keyword to search for products on Lidl's website. This initiates the scraping process for relevant items.	"laptop"
proxyConfiguration	object	No	Configures proxy settings to mask the scraper's origin and avoid detection. Defaults to using Apify's residential proxies.	{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}

Example Usage

To run the Lidl Category Scraper, provide the input parameters in JSON format. Here's an example:

{
  "searchKeyword": "laptop",
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

This will scrape products matching the keyword "laptop". The output is a JSON array of product objects, such as:

[
  {
    "productId": 100397341,
    "code": "100397341",
    "title": "HP Notebook »15-fd0611ng«, 15,6\" (39,6 cm)",
    "url": "https://www.lidl.de/p/hp-notebook-15-fd0611ng-15-6-39-6-cm/p100397341",
    "price": 249,
    "currency": "EUR",
    "originalPrice": 299,
    "discountPercent": 16,
    "hasDiscount": true,
    "image": "https://www.lidl.de/assets/gcpf2387dadc662435f8477494a24418aac.jpg",
    "images": [
      "https://www.lidl.de/assets/gcpf2387dadc662435f8477494a24418aac.jpg",
      "https://www.lidl.de/assets/gcpaaab8fe8c5884155852ab51881c149f5.jpg",
      "https://www.lidl.de/assets/gcp1b90584109a146099962680c86bfc0db.jpg"
    ],
    "brand": "HP",
    "category": "Kategorien/Multimedia/Computer & Büro/Laptops & Notebooks/Einsteiger Laptops",
    "onlineAvailable": true,
    "availabilityBadge": "auch in der Filiale von 24.11. - 29.11.",
    "description": "",
    "keyFacts": "",
    "supplementalInfo": "Bildschirmdiagonale: 15,6 ″\n Betriebssystem: Windows 11\n Prozessor: Intel® N100",
    "ageRestriction": false,
    "isUsedProduct": false,
    "dealOfDay": false,
    "flashSale": false,
    "energyLabelUrl": null,
    "relevancyScore": 1.785030484199524,
    "semanticScore": 0.9155159,
    "impressions": 2074,
    "clicks": 322,
    "carts": 41
  }
]

Use Cases

• Market Research and Analysis: Analyze product trends, pricing strategies, and competitor offerings in the German retail market.
• Competitive Intelligence: Monitor Lidl's product catalog for new arrivals, discounts, and category expansions to inform business decisions.
• Price Monitoring: Track price changes and discounts on specific items for dynamic pricing strategies or personal shopping alerts.
• Content Aggregation: Collect product data for e-commerce platforms, blogs, or apps requiring up-to-date retail information.
• Academic Research: Gather data on consumer goods for studies in economics, marketing, or supply chain management.
• Business Automation: Integrate with workflows to automate inventory checks, report generation, or notification systems for sales teams.

Installation and Usage

1. Search for "Lidl Category Scraper" in the Apify Store.
2. Click "Try for free" or "Run".
3. Configure input parameters (e.g., enter a search keyword).
4. Click "Start" to begin extraction.
5. Monitor progress in the log.
6. Export results in your preferred format (JSON, CSV, Excel).

Output Format

The Actor outputs a JSON array of objects, each representing a product. Key fields include:

• productId and code: Unique identifiers.
• title, url, price, currency: Basic product details.
• originalPrice, discountPercent, hasDiscount: Pricing and discount information.
• image, images: Product visuals.
• brand, category: Classification data.
• onlineAvailable, availabilityBadge: Stock status.
• description, keyFacts, supplementalInfo: Additional product info.
• ageRestriction, isUsedProduct, dealOfDay, flashSale: Special flags.
• relevancyScore, semanticScore, impressions, clicks, carts: Performance metrics.

Data is structured for easy parsing and integration with analytics tools.

Support

For custom/simplified outputs or bug reports, please contact:

• Email: support@getdataforme.com
• Subject line: "custom support"
• Contact form: https://getdataforme.com/contact/

We're here to help you get the most out of this Actor!