# Changelog of Lidl.de Products Scraper (`stone_jadex/lidl-de-products-scraper`) Actor

- **URL**: https://apify.com/stone_jadex/lidl-de-products-scraper/changelog.md
- **Full Actor documentation**: https://apify.com/stone_jadex/lidl-de-products-scraper.md

## Changelog

### 0.2.2 (review fixes for 422dafb)

- Status message: completed, truncated and no-match each get their own setStatusMessage line with its own try/catch and its own tests (incl. a failing status API per path). Mutation logs: deleting each line, or its try/catch, turns its test red (red: redlogs/mutations/status-completed, status-truncated, status-no-match and the matching -trycatch logs; green: redlogs/50).
- Pay-per-event check: the `product` event is charged exactly once per pushed row and never for no-match runs or filtered products; verified, no code change needed. New test asserts charge count = rows; the red log is a mutation run with a double charge per row (red: redlogs/49).
- `name` is again exactly the page title text as shown (only whitespace trimmed and the " | Lidl" site suffix dropped): a repeated brand, an uppercase brand such as GÜDE and brand letter case such as JUNGE WINZER are kept. Lidl's own cleaned title moves to the NEW field `nameClean` (84 product fields). Fixtures for p100003299, p100026080, p100018075, p100012177 (red: redlogs/48).
- Run status message: Actor.setStatusMessage (terminal, wrapped in try/catch) for every end state: completed with the row count, stopped early with the reason, no matches with a notice, no changes, failed (red: redlogs/47).

### 0.2.1

- README Known limitations and docs/COST.md: at most 1 (not 3) charged-but-unpushed row can be re-charged; rows are charged and pushed one at a time (red: redlogs/46).
- Test B1 (readme-review-d3d3a9f) regex follows the README wording that now also lists `" "` and `[]`; intent unchanged (red: redlogs/45).
- README Known limitations: rows whose dataset write fails 3 times go to OUTPUT.chargedButNotPushed and are charged again on a rerun or resumed run (red: redlogs/44).
- README: `" "` (whitespace only) and `[]` listed among num_items_found values for OUTPUT.searchFallbackQueries, verified against the parser (red: redlogs/43).
- README: `false` listed among num_items_found values that go to OUTPUT.searchFallbackQueries (red: redlogs/42).
- OUTPUT.notes text: robots.txt is warn-only (logged, counted in OUTPUT.robotsDisallowedHits, request proceeds), no longer claims enforcement; no field renamed or removed (red: redlogs/41).
- Input schema: incrementalStoreName gets editor "textfield" (platform build rejected it); test validates the schema with @apify/input_schema (red: redlogs/40).
- Stop rules (10-26): only entry/structure problems FAIL (empty sitemap, product pages without product data). Every product page 403/429, 5xx or network errors now end SUCCEEDED with one free notice row, nothing charged.

### 0.2.0 (review fixes for 5b3b2ea)

- Empty results: sitemap 200 with 0 products and "every product page 403/429" now FAIL the run (nothing charged); a filter that matches nothing, or no changes in incremental mode, SUCCEEDS with one free notice row; robots-unavailable is a non-failing protective stop.
- One test per safeguard; `node tools/mutate.mjs` deletes each safeguard (A–T) in a temp copy and proves a test turns red. Logs: redlogs/mutations/.
- Parser: full HTML5 entity decoding; GTIN check-digit validation (raw values stay in `eans`); description falls back to the long description.
- Proxy: custom proxyUrls refused; only direct or Apify Proxy non-residential groups.
- New fields brandLogoUrl, brandUrl (83 product fields). regionsPrices, zones, awards, infoTabs, FX conversions and lifecycle timestamps documented as unavailable.
- Competitor coverage checked against blackfalcondata/lidl-scraper's 42 published fields (docs/competitor-fields.json, sources cited).
- Cost per 1,000 rows documented with its calculation (docs/COST.md).
- README: competitor comparison for 1,000 rows, accurate EAN wording, runtime and limits for a full-catalogue run.

### 0.1.0 (2026-10-26)

- First version: Lidl.de product sitemap -> product pages (JSON-LD + embedded Nuxt data), HTTP only.
- Pay per event `product` $0.001, charge before push, push retries never re-charge, `chargedButNotPushed` in OUTPUT; run-notice, 404/410, filtered and unchanged products free.
- robots.txt first (5xx/unreachable = stop), allow-list + robots check on every redirect hop, ajv validation of every row.
- Filters: URL keywords, only orderable, only discounted, incremental (new or changed price/stock), compact rows.
- README: which num_items_found values go to OUTPUT.searchFallbackQueries (0, "0", null, empty) vs OUTPUT.searchFoundCountInvalid (non-numeric, missing) (red: redlogs/36).
- README Known limitations: hard kill + restart can re-charge up to about 99 rows (STATE_SAVE_EVERY = 100) (red: redlogs/37).
- README: 'Is it legal to scrape Lidl?' promoted to its own ## section; README tests updated (red: redlogs/38).
- Pricing decided: single per-product price, no per-keyword/search-page charge, worst-case $0.0027/run loss accepted (COST.md); README pricing says keywords with no results are not charged; no price value changed (red: redlogs/39).