Open Data Feed Watcher (Socrata, ArcGIS, CKAN)

Monitors open data datasets and portals using metadata only — it never downloads row data — and compares each run with the previous snapshot.

• Flags: stale data (vs stated frequency × grace factor), columns added/removed/renamed/retyped, row-count change above a threshold (default 10%), licence and title changes, CKAN resources added/removed, endpoint errors, column names that may hold personal info.
• Output: one JSON/CSV record per dataset or portal check: portal, dataset ID, status, HTTP status, age in days, row count and % change, licence, changes, PII-column hints.

Why not build it yourself? No per-platform metadata clients, snapshot storage or diff logic to write and maintain.

Price: $0.005 per check record (half a cent), no start fee. 20 datasets daily ≈ $0.10/day, about $3/month.

What it reports, per dataset

• Stale: the last update is older than the stated frequency allows (frequency × staleGraceFactor, default 1.5). You can set maxAgeDays per dataset, which you need for ArcGIS because it states no frequency.
• Columns added, removed, renamed or retyped. Renames are matched by Socrata column id, or else by a single removal and addition of the same type (sameTypeHeuristic).
• Row count change above rowCountThresholdPct (default 10%).
• Licence and title changes. CKAN resources added or removed.
• Endpoint errors: HTTP status or API error, and recovery on the next good run.
• Possible personal info (warning only): column names such as name, email, phone, address, DOB, owner, SIN, VIN or lat/long. Column contents are never read.

Whole-portal mode

Under portals, give {"portal": "socrata"|"ckan"|"arcgis", "domain" or "url", "maxDatasets"}. Each run outputs one catalogue record listing the datasets added, removed or retitled since the last run. ArcGIS lists the services under a /rest/services URL.

Metadata endpoints used

Portal	Calls
Socrata	/api/views/<id>.json, plus /api/catalog/v1 for portal scans
ArcGIS	layer ?f=json, query?where=1=1&returnCountOnly=true
CKAN	package_show, package_search (portal scan), and datastore_search?limit=0 (field list and total only, optional)

Politeness

It reads robots.txt for each host and honours Crawl-delay, with a minimum of 1 s between calls to a host. open.canada.ca asks for 20 s; that is also a built-in fallback, because its robots.txt can block plain clients. It sends a normal User-Agent and retries 429 and 5xx responses with backoff.

Input example

{
  "datasets": [
    {"portal": "socrata", "domain": "data.novascotia.ca", "id": "m6ps-8j6u"},
    {"portal": "arcgis", "url": "https://services2.arcgis.com/11XBiaBYA9Ep0yNJ/arcgis/rest/services/PPLC_Issued_Building_Permits/FeatureServer/0", "maxAgeDays": 14},
    {"portal": "ckan", "domain": "open.canada.ca", "id": "0032ce54-c5dd-4b66-99a0-320a7b5e99f2"}
  ],
  "portals": [{"portal": "socrata", "domain": "data.novascotia.ca", "maxDatasets": 2000}],
  "rowCountThresholdPct": 10,
  "onlyChanges": false
}

Use a different snapshotKey for each watch list. On the first run every dataset has firstSeen: true and no diffs.

Development

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt pytest
.venv/bin/python -m pytest -q
.venv/bin/python -m src     # input in storage/key_value_stores/default/INPUT.json

Pricing

$0.005 per result (pay per event, one result = one dataset or portal check record), with no start fee. Watching 20 datasets daily costs about $0.10 a day, roughly $3 a month.