EU Pesticide Active Substance Status Watch

Marketplace-native Apify Actor that watches the official EU pesticide active substances register (Regulation (EC) No 1107/2009) through the European Commission's own data infrastructure and turns it into a structured, provenance-linked change feed:

• Approval-status changes (Approved ⇄ Not approved ⇄ Pending) detected by hash-based diffing of the complete universe (~4,700 substances) between runs;
• Expiry-date changes with exact before/after values;
• Informational countdown clocks to officially published approval-expiry dates, with T-365 / T-180 / T-90 / T-30 threshold flags and an optional expiring-soon watch window;
• CELEX provenance: every change record carries the implementing regulations linked to the substance, extracted from the official source;
• Stable IDs and a run summary heartbeat so downstream systems can dedupe and audit.

Disclaimer (applies to every output record). This Actor reports official EU source data as published at run time plus pure date arithmetic on it. It is informational only — not legal, regulatory, compliance, scientific or financial advice. It does not interpret Article 46 grace periods, sell-off, use-up or disposal windows; those require legal analysis of each implementing regulation. Always verify against the Official Journal before making decisions.

Data sources & attribution

What	Where
Primary dataset	https://api.datalake.sante.service.ec.europa.eu/sante/pesticides/active-substances-download?format=json&api-version=v3.0
Publisher	European Commission, Directorate-General for Health and Food Safety (DG SANTE)
Same data as	EU Pesticides Database
Update cadence of the source	Daily (per DG SANTE "Pesticides – APIs V3.0" documentation)
Licence	CC BY 4.0 — re-use of Commission information under Decision 2011/833/EU

Every emitted record includes an attribution, the exact source URLs used, retrieval timestamp, API version, licence note and the disclaimer above. The Actor performs one polite GET request per run to the official endpoint; no authentication is required and no access controls are circumvented.

How it works

1. Load bounded incremental state from the default key-value store (one JSON document, ~1 MB ceiling; corrupt or incompatible state degrades safely to a fresh baseline).
2. Fetch the official full-dataset export (single request with retries, timeouts, jittered backoff and Retry-After support).
3. Parse NDJSON leniently; abort rather than emit if malformed-line ratio exceeds 2% or valid records fall below the integrity floor.
4. Normalize all substances and diff against stored tracked fields (11 fields incl. status and expiry date). Filters are a pure output view — detection always covers the whole universe.
5. Emit, in deterministic order and capped by maxItems: run_summary → substance_change records → expiry_watch countdowns → substance_snapshot rows (full mode).
6. Only after successful dataset writes, atomically commit the new state snapshot to the named key-value store eu-pesticide-substance-watch-state (key state). A named store is required because the platform's default store is scoped per run.

Dataset record types

type	Meaning
run_summary	Always present exactly once: mode, baseline flag, universe size, per-kind change counts, integrity counters, state size.
substance_change	new_substance, removed_substance, status_change, expiry_date_change or other_field_change, with changes[] before/after pairs, countdown, CELEX links, remark.
expiry_watch	Approved substances whose published expiry falls within your window; sorted most-urgent-first with crossed thresholds.
substance_snapshot	Full-mode filtered view of current official status incl. daysRemaining.

Record IDs are stable SHA-1 digests of business content (no timestamps), so re-runs over identical data are idempotent for consumers. checkedAt carries the run time separately.

Input

Field	Default	Notes
mode	incremental	full additionally emits filtered snapshots (recommended first run).
statuses	[] (all)	Any of Approved, Not approved, Pending. Output filter only.
nameQuery	""	Case-insensitive substring on the official name.
expiringWithinDays	0 (off)	Emits expiry_watch for Approved expiring within N days.
countdownThresholdsDays	[365,180,90,30]	Threshold flags attached to countdowns.
maxItems	5000	Hard cap on per-substance records per run (summary excluded).
includeToxicology	false	Adds official ADI value/source to snapshots.

The example run input (prefill) uses mode=full, statuses=["Approved"], maxItems=300 — a bounded, real-source run that completes well within five minutes and always writes a non-empty dataset.

Usage

Run in Apify Console or via CLI:

apify run --purge          # local, uses storage/key_value_stores/default/INPUT.json
apify push                 # deploy to Apify cloud

Consume the results:

$curl -s "https://api.apify.com/v2/datasets/<DATASET_ID>/items?clean=true" | jq '.[0:5]'

Typical schedule: daily or weekly incremental runs; a weekly full run as a self-audit baseline. Between regulatory events, keep expiringWithinDays set (e.g. 540) so the Actor still delivers a standing countdown view even when nothing changed.

Known limits (explicit unknowns)

• Source dates are reported exactly as published; some substances carry empty/null dates.
• Removal detection trusts the completeness of the upstream export; a >10% mass disappearance is treated as an upstream failure and refused instead of reported.
• UK HSE (GB) divergence tracking and EUR-Lex OJ monitoring are not part of this version.
• Countdowns are arithmetic on official expiry dates only; member-state divergences, grace periods and cancellations are out of scope and must be verified with counsel.

Development

npm install
npm run lint && npm run typecheck && npm run build
npm test                  # unit + integration + adversarial + schema (offline)
npm run test:smoke        # live-source smoke against the official API (SMOKE=1)
npm run smoke:local       # full local Actor run with example input

Node.js ≥ 20. TypeScript strict mode; ESLint flat config. Tests use the Node built-in runner via tsx; integration/adversarial suites run fully offline against fixtures and a local HTTP server.