Public Lidl.de online-shop products from the official sitemap: price, struck-through price and discount, unit price, EAN/GTIN, brand, category, stock, rating, images, attributes, variants. $1.00 per 1,000 products, no start fee. HTTP only, robots.txt respected.
Status message: completed, truncated and no-match each get their own setStatusMessage line with its own try/catch and its own tests (incl. a failing status API per path). Mutation logs: deleting each line, or its try/catch, turns its test red (red: redlogs/mutations/status-completed, status-truncated, status-no-match and the matching -trycatch logs; green: redlogs/50).
Pay-per-event check: the product event is charged exactly once per pushed row and never for no-match runs or filtered products; verified, no code change needed. New test asserts charge count = rows; the red log is a mutation run with a double charge per row (red: redlogs/49).
name is again exactly the page title text as shown (only whitespace trimmed and the " | Lidl" site suffix dropped): a repeated brand, an uppercase brand such as GÜDE and brand letter case such as JUNGE WINZER are kept. Lidl's own cleaned title moves to the NEW field nameClean (84 product fields). Fixtures for p100003299, p100026080, p100018075, p100012177 (red: redlogs/48).
Run status message: Actor.setStatusMessage (terminal, wrapped in try/catch) for every end state: completed with the row count, stopped early with the reason, no matches with a notice, no changes, failed (red: redlogs/47).
0.2.1
README Known limitations and docs/COST.md: at most 1 (not 3) charged-but-unpushed row can be re-charged; rows are charged and pushed one at a time (red: redlogs/46).
Test B1 (readme-review-d3d3a9f) regex follows the README wording that now also lists " " and []; intent unchanged (red: redlogs/45).
README Known limitations: rows whose dataset write fails 3 times go to OUTPUT.chargedButNotPushed and are charged again on a rerun or resumed run (red: redlogs/44).
README: " " (whitespace only) and [] listed among num_items_found values for OUTPUT.searchFallbackQueries, verified against the parser (red: redlogs/43).
README: false listed among num_items_found values that go to OUTPUT.searchFallbackQueries (red: redlogs/42).
OUTPUT.notes text: robots.txt is warn-only (logged, counted in OUTPUT.robotsDisallowedHits, request proceeds), no longer claims enforcement; no field renamed or removed (red: redlogs/41).
Input schema: incrementalStoreName gets editor "textfield" (platform build rejected it); test validates the schema with @apify/input_schema (red: redlogs/40).
Stop rules (10-26): only entry/structure problems FAIL (empty sitemap, product pages without product data). Every product page 403/429, 5xx or network errors now end SUCCEEDED with one free notice row, nothing charged.
0.2.0 (review fixes for 5b3b2ea)
Empty results: sitemap 200 with 0 products and "every product page 403/429" now FAIL the run (nothing charged); a filter that matches nothing, or no changes in incremental mode, SUCCEEDS with one free notice row; robots-unavailable is a non-failing protective stop.
One test per safeguard; node tools/mutate.mjs deletes each safeguard (A–T) in a temp copy and proves a test turns red. Logs: redlogs/mutations/.
Parser: full HTML5 entity decoding; GTIN check-digit validation (raw values stay in eans); description falls back to the long description.
Proxy: custom proxyUrls refused; only direct or Apify Proxy non-residential groups.
New fields brandLogoUrl, brandUrl (83 product fields). regionsPrices, zones, awards, infoTabs, FX conversions and lifecycle timestamps documented as unavailable.
Competitor coverage checked against blackfalcondata/lidl-scraper's 42 published fields (docs/competitor-fields.json, sources cited).
Cost per 1,000 rows documented with its calculation (docs/COST.md).
README: competitor comparison for 1,000 rows, accurate EAN wording, runtime and limits for a full-catalogue run.
Pay per event product $0.001, charge before push, push retries never re-charge, chargedButNotPushed in OUTPUT; run-notice, 404/410, filtered and unchanged products free.
robots.txt first (5xx/unreachable = stop), allow-list + robots check on every redirect hop, ajv validation of every row.
Filters: URL keywords, only orderable, only discounted, incremental (new or changed price/stock), compact rows.
README: which num_items_found values go to OUTPUT.searchFallbackQueries (0, "0", null, empty) vs OUTPUT.searchFoundCountInvalid (non-numeric, missing) (red: redlogs/36).
README Known limitations: hard kill + restart can re-charge up to about 99 rows (STATE_SAVE_EVERY = 100) (red: redlogs/37).
README: 'Is it legal to scrape Lidl?' promoted to its own ## section; README tests updated (red: redlogs/38).
Pricing decided: single per-product price, no per-keyword/search-page charge, worst-case $0.0027/run loss accepted (COST.md); README pricing says keywords with no results are not charged; no price value changed (red: redlogs/39).
