# EU Pesticide Active Substance Status Watch (`atlas-data/eu-pesticide-substance-watch`) Actor

Monitors the official EU pesticide active substances register (Regulation 1107/2009) via the European Commission's DG SANTE API. Detects approval-status and expiry-date changes since the last run, computes informational expiry countdowns, and emits CELEX-provenance-linked records.

- **URL**: https://apify.com/atlas-data/eu-pesticide-substance-watch.md
- **Developed by:** [Atlas](https://apify.com/atlas-data) (community)
- **Categories:** Business
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## EU Pesticide Active Substance Status Watch

Marketplace-native Apify Actor that watches the **official EU pesticide active substances register**
(Regulation (EC) No 1107/2009) through the European Commission's own data infrastructure and turns
it into a structured, provenance-linked change feed:

- **Approval-status changes** (`Approved` ⇄ `Not approved` ⇄ `Pending`) detected by hash-based
  diffing of the complete universe (~4,700 substances) between runs;
- **Expiry-date changes** with exact before/after values;
- **Informational countdown clocks** to officially published approval-expiry dates, with T-365 /
  T-180 / T-90 / T-30 threshold flags and an optional *expiring-soon* watch window;
- **CELEX provenance**: every change record carries the implementing regulations linked to the
  substance, extracted from the official source;
- **Stable IDs** and a run summary heartbeat so downstream systems can dedupe and audit.

> **Disclaimer (applies to every output record).** This Actor reports official EU source data as
> published at run time plus pure date arithmetic on it. It is **informational only — not legal,
> regulatory, compliance, scientific or financial advice**. It does **not** interpret Article 46
> grace periods, sell-off, use-up or disposal windows; those require legal analysis of each
> implementing regulation. Always verify against the **Official Journal** before making decisions.

### Data sources & attribution

| What | Where |
|---|---|
| Primary dataset | `https://api.datalake.sante.service.ec.europa.eu/sante/pesticides/active-substances-download?format=json&api-version=v3.0` |
| Publisher | European Commission, Directorate-General for Health and Food Safety (DG SANTE) |
| Same data as | [EU Pesticides Database](https://food.ec.europa.eu/plants/pesticides/eu-pesticides-database_en) |
| Update cadence of the source | Daily (per DG SANTE "Pesticides – APIs V3.0" documentation) |
| Licence | [CC BY 4.0](https://spdx.org/licenses/CC-BY-4.0) — re-use of Commission information under Decision 2011/833/EU |

Every emitted record includes an `attribution`, the exact source URLs used, retrieval timestamp,
API version, licence note and the disclaimer above. The Actor performs **one polite GET request**
per run to the official endpoint; no authentication is required and no access controls are
circumvented.

### How it works

1. Load bounded incremental state from the default key-value store (one JSON document,
   \~1 MB ceiling; corrupt or incompatible state degrades safely to a fresh baseline).
2. Fetch the official full-dataset export (single request with retries, timeouts, jittered
   backoff and `Retry-After` support).
3. Parse NDJSON leniently; abort rather than emit if malformed-line ratio exceeds 2% or valid
   records fall below the integrity floor.
4. Normalize all substances and diff against stored tracked fields (11 fields incl. status and
   expiry date). Filters are a pure output view — detection always covers the whole universe.
5. Emit, in deterministic order and capped by `maxItems`: `run_summary` → `substance_change`
   records → `expiry_watch` countdowns → `substance_snapshot` rows (full mode).
6. Only after successful dataset writes, atomically commit the new state snapshot to the
   **named key-value store** `eu-pesticide-substance-watch-state` (key `state`). A named store is
   required because the platform's *default* store is scoped per run.

#### Dataset record types

| `type` | Meaning |
|---|---|
| `run_summary` | Always present exactly once: mode, baseline flag, universe size, per-kind change counts, integrity counters, state size. |
| `substance_change` | `new_substance`, `removed_substance`, `status_change`, `expiry_date_change` or `other_field_change`, with `changes[]` before/after pairs, countdown, CELEX links, remark. |
| `expiry_watch` | Approved substances whose published expiry falls within your window; sorted most-urgent-first with crossed thresholds. |
| `substance_snapshot` | Full-mode filtered view of current official status incl. `daysRemaining`. |

Record IDs are stable SHA-1 digests of business content (no timestamps), so re-runs over identical
data are idempotent for consumers. `checkedAt` carries the run time separately.

### Input

| Field | Default | Notes |
|---|---|---|
| `mode` | `incremental` | `full` additionally emits filtered snapshots (recommended first run). |
| `statuses` | `[]` (all) | Any of `Approved`, `Not approved`, `Pending`. Output filter only. |
| `nameQuery` | `""` | Case-insensitive substring on the official name. |
| `expiringWithinDays` | `0` (off) | Emits `expiry_watch` for Approved expiring within N days. |
| `countdownThresholdsDays` | `[365,180,90,30]` | Threshold flags attached to countdowns. |
| `maxItems` | `5000` | Hard cap on per-substance records per run (summary excluded). |
| `includeToxicology` | `false` | Adds official ADI value/source to snapshots. |

The example run input (prefill) uses `mode=full`, `statuses=["Approved"]`, `maxItems=300` — a
bounded, real-source run that completes well within five minutes and always writes a non-empty
dataset.

### Usage

Run in Apify Console or via CLI:

```bash
apify run --purge          # local, uses storage/key_value_stores/default/INPUT.json
apify push                 # deploy to Apify cloud
```

Consume the results:

```bash
curl -s "https://api.apify.com/v2/datasets/<DATASET_ID>/items?clean=true" | jq '.[0:5]'
```

Typical schedule: daily or weekly incremental runs; a weekly `full` run as a self-audit baseline.
Between regulatory events, keep `expiringWithinDays` set (e.g. 540) so the Actor still delivers a
standing countdown view even when nothing changed.

### Known limits (explicit unknowns)

- Source dates are reported exactly as published; some substances carry empty/null dates.
- Removal detection trusts the completeness of the upstream export; a >10% mass disappearance is
  treated as an upstream failure and refused instead of reported.
- UK HSE (GB) divergence tracking and EUR-Lex OJ monitoring are **not** part of this version.
- Countdowns are arithmetic on official expiry dates only; member-state divergences, grace
  periods and cancellations are out of scope and must be verified with counsel.

### Development

```bash
npm install
npm run lint && npm run typecheck && npm run build
npm test                  # unit + integration + adversarial + schema (offline)
npm run test:smoke        # live-source smoke against the official API (SMOKE=1)
npm run smoke:local       # full local Actor run with example input
```

Node.js ≥ 20. TypeScript strict mode; ESLint flat config. Tests use the Node built-in runner via
`tsx`; integration/adversarial suites run fully offline against fixtures and a local HTTP server.

# Actor input Schema

## `mode` (type: `string`):

incremental: emits change records for substances whose tracked fields changed since the previous run, plus optional expiry-watch records. full: additionally emits one snapshot record per substance matching the filters (useful for the first run or periodic rebuilds). Change detection always covers the complete universe in both modes.

## `statuses` (type: `array`):

Only substances with these official statuses are emitted (change detection still runs on all substances so nothing is missed). Leave empty to include every status.

## `nameQuery` (type: `string`):

Case-insensitive substring match against the official substance name (e.g. 'chlorpyrifos'). Empty disables the filter.

## `expiringWithinDays` (type: `integer`):

Emits expiry\_watch records for Approved substances whose officially published approval expiry date falls within this many days. 0 disables the watch. Countdowns are informational arithmetic on official dates — they are not legal advice and do not interpret sell-off or use-up grace periods.

## `countdownThresholdsDays` (type: `array`):

Threshold flags attached to countdowns (e.g. T-365, T-180). Integers between 1 and 3650.

## `maxItems` (type: `integer`):

Hard cap on per-substance dataset records emitted in one run (the run summary is always written first; state always covers the whole universe regardless of this cap).

## `includeToxicology` (type: `boolean`):

Adds the officially published Acceptable Daily Intake (ADI) value and its source regulation to full-mode snapshots.

## Actor input object example

```json
{
  "mode": "full",
  "statuses": [
    "Approved"
  ],
  "nameQuery": "",
  "expiringWithinDays": 540,
  "countdownThresholdsDays": [
    365,
    180,
    90,
    30
  ],
  "maxItems": 300,
  "includeToxicology": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "full",
    "statuses": [
        "Approved"
    ],
    "nameQuery": "",
    "expiringWithinDays": 540,
    "countdownThresholdsDays": [
        365,
        180,
        90,
        30
    ],
    "maxItems": 300,
    "includeToxicology": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("atlas-data/eu-pesticide-substance-watch").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "full",
    "statuses": ["Approved"],
    "nameQuery": "",
    "expiringWithinDays": 540,
    "countdownThresholdsDays": [
        365,
        180,
        90,
        30,
    ],
    "maxItems": 300,
    "includeToxicology": False,
}

# Run the Actor and wait for it to finish
run = client.actor("atlas-data/eu-pesticide-substance-watch").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "full",
  "statuses": [
    "Approved"
  ],
  "nameQuery": "",
  "expiringWithinDays": 540,
  "countdownThresholdsDays": [
    365,
    180,
    90,
    30
  ],
  "maxItems": 300,
  "includeToxicology": false
}' |
apify call atlas-data/eu-pesticide-substance-watch --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,atlas-data/eu-pesticide-substance-watch"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Kg1ImXfJahzrmA6Ms/builds/L4Hq3j2EMkFTtkpVA/openapi.json