# Lidl Scraper & Weekly Flyers \[8 Countries • $2.50] (`silver_petalfall/lidl-catalog-price-scraper`) Actor

Scrape Lidl.es (Spain) and 7 EU markets (lidl.de, lidl.fr, lidl.co.uk, lidl.it, lidl.pt, lidl.pl, lidl.nl). Extract 100% verified shelf prices, weekly flyer offers, groceries, photos & EAN codes in one unified dataset. Fast Pay-Per-Event ($2.50/1k).

- **URL**: https://apify.com/silver_petalfall/lidl-catalog-price-scraper.md
- **Developed by:** [Javi Face](https://apify.com/silver_petalfall) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $2.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 🛒 Lidl Scraper & Weekly Flyers (8 European Countries)

#### Extract Verified In-Store Leaflets, Supermarket Groceries & Online Catalog with 100% Shelf Prices

[![Apify Actor](https://img.shields.io/badge/Apify-Actor-FF9000?style=for-the-badge\&logo=apify\&logoColor=white)](https://apify.com) [![Pay Per Event](https://img.shields.io/badge/Pricing-Pay--Per--Event%20\($2.50%2F1k\)-2ECC71?style=for-the-badge)](https://apify.com) [![Speed](https://img.shields.io/badge/Speed-100%20items%20%2F%209s-0050AA?style=for-the-badge)](https://apify.com) [![Coverage](https://img.shields.io/badge/Markets-8%20EU%20Countries-E60A14?style=for-the-badge)](#market-coverage) [![Data Quality](https://img.shields.io/badge/Shelf%20Prices-100%25%20Verified-F1C40F?style=for-the-badge)](#commercial-differentiation)

> **High-speed European retail intelligence engine.** Automatically extract verified in-store weekly leaflet promotions, supermarket grocery assortments, and online catalog pricing across 8 European countries into a standardized, unified schema.

***

### 🎯 Commercial Differentiation (B2B Value)

Most retail scrapers rely on heavy, resource-intensive browser emulation that fails when querying grocery inventories or gets blocked by perimeter firewalls. Other tools pull stale, crowdsourced data with missing prices and restrictive open-database licenses.

This Actor implements a **direct, first-party intelligence pipeline** that delivers 100% verified shelf prices with zero third-party contamination:

| Capability                                                |        Generic Scrapers        |                **Lidl Enterprise Intelligence (This Actor)**                |
| :-------------------------------------------------------- | :----------------------------: | :-------------------------------------------------------------------------: |
| **In-Store Weekly Promotions** (Leaflets & Special Deals) |         ❌ Unavailable         |   **✅ Full Coverage (Promotional dates, discount %, and HD page crops)**   |
| **Supermarket Assortment** (Food, Cleaning, Cosmetics)    |    ❌ Incomplete / Blocked     | **✅ Comprehensive (Fresh food, private labels, pack sizes, shelf prices)** |
| **Price Data Completeness**                               |      ⚠️ Frequently `null`      |            **✅ 100% Verified Shelf Prices (Zero null prices)**             |
| **Anti-Bot Resilience**                                   |    ❌ Repeated HTTP 403/429    |       **✅ Advanced Browser Fingerprint Spoofing & Session Rotation**       |
| **Infrastructure Overhead**                               |      ⚠️ Heavy (2GB+ RAM)       |           **✅ Ultra-Lightweight (256MB – 512MB RAM container)**            |
| **Data Licensing**                                        | ⚠️ Restrictive Copyleft (ODbL) |        **✅ 100% First-Party Data (Clean for B2B & Commercial Use)**        |
| **Visual Validation**                                     |    ❌ Raw text / JSON only     |      **✅ Built-in Visual Table with High-Resolution Product Photos**       |

***

### 🌍 Market Coverage

Seamlessly toggle between 8 regional retail operations with zero configuration:

|         Market          | Domain       | Primary Currency | Assortment Scope                                      |
| :---------------------: | :----------- | :--------------: | :---------------------------------------------------- |
|     **Spain (ES)**      | `lidl.es`    |     EUR (€)      | Weekly Leaflets, Supermarket Assortment & Online Shop |
|    **Germany (DE)**     | `lidl.de`    |     EUR (€)      | Weekly Leaflets, Supermarket Assortment & Online Shop |
|     **France (FR)**     | `lidl.fr`    |     EUR (€)      | Weekly Leaflets, Supermarket Assortment & Online Shop |
| **United Kingdom (GB)** | `lidl.co.uk` |     GBP (£)      | Weekly Leaflets & Supermarket Assortment              |
|     **Italy (IT)**      | `lidl.it`    |     EUR (€)      | Weekly Leaflets & Supermarket Assortment              |
|    **Portugal (PT)**    | `lidl.pt`    |     EUR (€)      | Weekly Leaflets & Supermarket Assortment              |
|     **Poland (PL)**     | `lidl.pl`    |     PLN (zł)     | Weekly Leaflets, Supermarket Assortment & Online Shop |
|  **Netherlands (NL)**   | `lidl.nl`    |     EUR (€)      | Weekly Leaflets, Supermarket Assortment & Online Shop |

***

### 🛡️ Enterprise-Grade Architecture & Resilience

- **Advanced Browser Fingerprint Spoofing**: Proprietary client TLS layer mimics genuine browser connection characteristics to bypass perimeter traffic inspection without the latency and memory overhead of headless browsers.
- **Intelligent Session Management**: Automatic proxy session rotation paired with adaptive exponential backoff prevents IP reputation degradation and ensures continuous throughput.
- **Strict Data Integrity Contracts**: Every extracted data point passes through schema-level sanitization (decimal normalization, HTML stripping, automated discount percentage calculation).
- **Enterprise Security & Zero PII**: Strict URL allowlisting protects against SSRF vulnerabilities. Extracts strictly public product catalog data; zero customer reviews, personal names, or account data are ever processed (GDPR compliant).

***

### ⚡ Zero-Config Quick Start

1. Open this Actor on the Apify Console and click **Start**.
2. By default, it extracts a balanced sample of **100 products** and **10 weekly flyer pages** with verified prices, discounts, package sizes, and high-resolution photographs.
3. Export to **JSON**, **CSV**, or **Excel**, or inspect the live photo catalog in the **Visual Overview** tab.

#### Sample Input

```json
{
  "country": "ES",
  "categories": ["weekly_flyers", "food", "cleaning", "cosmetics"],
  "searchQueries": ["milbona", "cien", "w5"],
  "maxItems": 100
}
```

***

### 📊 Standardized Product Schema

All records are normalized into a clean, unified structure regardless of country:

```json
{
  "product_id": "11011151",
  "name": "MILBONA Yogur frutas con trozos",
  "brand": "MILBONA",
  "category_macro": "food",
  "subcategory": "Yogur",
  "price": 2.09,
  "original_price": 2.15,
  "discount_percentage": 2,
  "currency": "EUR",
  "unit_price_text": null,
  "package_size": "400gr",
  "availability": "IN_STORE",
  "country": "ES",
  "source_vector": "web_catalog",
  "product_url": "https://www.lidl.es/p/milbona-yogur-frutas-con-trozos/p11011151",
  "image_url": "https://www.lidl.es/assets/gcp2fd08fa4104447b6b69fa8f182c3002a.png",
  "additional_images": ["https://www.lidl.es/assets/gcp223b11909cbb43b48fd602de8983f526.png"],
  "valid_from": "2026-10-01",
  "valid_to": "2026-10-31",
  "scraped_at": "2026-10-15T09:31:21.567203+00:00"
}
```

***

### 💰 Pricing & Predictable Cost Control

This Actor uses Apify's **Pay-Per-Event (PPE)** pricing model. You only pay for verified data delivered to your dataset:

| Event                      | Price (USD) | Description                                                        |
| :------------------------- | :---------: | :----------------------------------------------------------------- |
| **`actor-start`**          |  **$0.01**  | One-time execution start fee covering container initialization.    |
| **`product-scraped`**      | **$0.0025** | Per unique validated product ($2.50 per 1,000 items).              |
| **`flyer-page-processed`** | **$0.005**  | Per processed promotional flyer page (HD scans, dates, hot-spots). |

> **Exact Budget Ceiling**: The Actor halts immediately upon reaching your `maxItems` or total budget limit. You are never billed beyond your configured maximum.

#### 💡 Example Cost Estimates

| Volume | Approximate Run Time | Total Estimated Cost |
| :--- | :---: | :---: |
| **Sample Test (100 products)** | **~9 seconds** | **$0.26** |
| **Category Scan (500 products)** | **~25 seconds** | **$1.26** |
| **Full Assortment (1,000 products)** | **~45 seconds** | **$2.51** |
| **Multi-Store Basket (5,000 products)** | **~3 minutes** | **$12.51** |

***

### 🤖 AI Agents & Integrations (MCP Ready)

Easily connect this Actor with modern AI automation and workflow pipelines:

- **Model Context Protocol (MCP)**: Native compatibility with Claude Code, Cursor, Antigravity, and AI Agents for autonomous market research.
- **Direct Export**: Instant 1-click download in **JSON**, **CSV**, or **Excel (.xlsx)**.
- **REST API & Webhooks**: Trigger downstream syncs whenever new weekly flyer deals are published.
- **Google Sheets**: Connect directly using Apify's API endpoint with `=IMPORTDATA()`.

***

### 💼 B2B & Enterprise Use Cases

- **Private Label Intelligence**: Monitor price elasticity, shrinkflation, and promotional cadences across proprietary store brands (Milbona, Cien, W5, Parkside).
- **Weekly Promotional Aggregators**: Seamlessly ingest in-store leaflet promotions, promotional dates, and catalog images into coupon or shopping comparison portals.
- **Dynamic Pricing Algorithms**: Feed clean, real-time retail price points directly into predictive pricing and revenue optimization models.
- **Inflation & Market Baskets**: Track price movements across identical consumer baskets in 8 European markets over time.

***

### ❓ Frequently Asked Questions (FAQ)

#### Can I scrape in-store weekly flyer deals that are not listed in the online shop?

**Yes.** Unlike basic web scrapers that only read the e-commerce storefront, this Actor directly captures the official in-store promotional leaflets with full promotional validity dates, discount percentages, and high-resolution scanned page crops.

#### Are prices 100% verified?

**Yes.** Every single item contains real shelf prices and discount amounts extracted directly from first-party store endpoints. You will never encounter `price: null` records.

#### How do I export data to Excel or Google Sheets?

After running the Actor, go to the **Storage** tab in Apify Console and select **Excel (.xlsx)** or **CSV**. You can also copy the API endpoint into Google Sheets.

#### Can I run this on a recurring schedule?

**Yes.** In the Apify Console, navigate to the **Schedules** tab to run this Actor automatically every week when new weekly flyer promotions go live (e.g. Mondays or Thursdays).

***

### 🤝 Custom Integrations & Dedicated Pipelines

Need custom data pipelines, high-volume dedicated scraping, or automated delivery to **Snowflake, BigQuery, AWS S3, or PostgreSQL**?

Reach out to our engineering team through the **Issues** or contact options on our Apify profile.

# Actor input Schema

## `country` (type: `string`):

Select the country store you want to extract products and prices from.

## `categories` (type: `array`):

Choose which sections of Lidl's catalog to include. <b>weekly_flyers</b> = in-store weekly leaflet offers & discounts; <b>food / cleaning / cosmetics</b> = supermarket groceries with verified shelf prices & photos; <b>bazaar_online</b> = non-food online shop.

## `searchQueries` (type: `array`):

Filter by specific terms or Lidl private brands (e.g., 'leche', 'detergente', 'milbona', 'cien', 'w5', 'parkside'). Leave empty to automatically use comprehensive built-in keywords with guaranteed real shelf prices.

## `maxItems` (type: `integer`):

Limit the total number of products to control your run cost. You are never charged above this limit.

## `maxFlyerPages` (type: `integer`):

How many weekly-flyer pages (HD image, PDF link, keywords) to save in the FLYER_PAGES record. Set 0 to only get flyer products.

## `fullCatalogMode` (type: `boolean`):

Crawl every product page listed in Lidl's product sitemap (JSON-LD). Slower but exhaustive. Ignores keywords.

## `startUrls` (type: `array`):

Direct Lidl product URLs (…/p/…), flyer URLs (…/l/…) or search URLs. Only official *.lidl.* domains are accepted.

## `proxyConfiguration` (type: `object`):

Datacenter proxies are enough in most cases. Switch to Residential only if you see repeated 403 errors.

## Actor input object example

```json
{
  "country": "ES",
  "categories": [
    "weekly_flyers",
    "food",
    "cleaning",
    "cosmetics"
  ],
  "searchQueries": [
    "milbona",
    "cien",
    "w5"
  ],
  "maxItems": 100,
  "maxFlyerPages": 10,
  "fullCatalogMode": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

Primary dataset containing verified grocery and catalog products with shelf prices, discounts, and high-res photos.

## `offers` (type: `string`):

Dataset view filtered for active promotions, flyer page references, and validity date ranges.

## `flyer_pages` (type: `string`):

Key-value store record containing in-store weekly flyer page images, PDF links, and promotion dates.

## `run_stats` (type: `string`):

Key-value store record containing execution summary, items counted, and country metrics.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "milbona",
        "cien",
        "w5"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("silver_petalfall/lidl-catalog-price-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": [
        "milbona",
        "cien",
        "w5",
    ],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("silver_petalfall/lidl-catalog-price-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "milbona",
    "cien",
    "w5"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call silver_petalfall/lidl-catalog-price-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,silver_petalfall/lidl-catalog-price-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/yqoI8oA3E82YZKPcn/builds/aELUSs1oUdbjcsmI2/openapi.json