CKAN & Socrata Schema Change Monitor

Detect broken or drifting government open-data feeds before they silently damage a report, integration, or archived snapshot.

This Actor checks bounded samples from explicit public CKAN and Socrata APIs and returns a source receipt for every endpoint. It is designed for scheduled monitoring, ETL checks, reproducibility records, and data-quality alerts.

What each check reports

• new, unchanged, changed, stale, or failed state
• observed versus source-reported row counts, kept distinct
• required-field violations with affected sample indexes
• duplicate compound-key count
• added, removed, and type-changed fields
• newest observed source timestamp and freshness
• deterministic schema, sample, and raw-response SHA-256 hashes
• request URL, HTTP status, ETag, Last-Modified, and retrieval time

Baselines are preserved across scheduled runs in a named Apify key-value store. Use a unique baselineStoreName for each independent monitor configuration.

Supported sources

Socrata

Provide a public domain and dataset ID such as data.cityofnewyork.us and erm2-nwe9. Optional query parameters may sort or filter the bounded sample. The Actor always enforces its own row cap.

CKAN

Provide an HTTPS CKAN action API base and either a package ID or search query. Package results are checked as tabular metadata records.

Boundaries

• Public HTTPS APIs only.
• No passwords, private headers, cookies, login automation, CAPTCHA handling, or access-control bypass.
• Local, loopback, link-local, and private-IP destinations are rejected before HTTP fetch.
• Maximum 50 sources and 100 observed rows per source per run.
• A bounded sample is never described as a complete-dataset count.
• A successful request is not called healthy when it returns zero rows or lacks the configured freshness field; warnings remain explicit.

Pricing

$0.05 per source-check event. One event produces one source receipt, including an explicit failure receipt when a configured public endpoint returns an HTTP or validation error. Apify platform usage is included in the event price.

Example input

{
  "maxRows": 10,
  "baselineStoreName": "city-open-data-daily",
  "sources": [
    {
      "kind": "socrata",
      "domain": "data.cityofnewyork.us",
      "datasetId": "erm2-nwe9",
      "requiredFields": ["unique_key", "created_date"],
      "keyFields": ["unique_key"],
      "freshnessField": "created_date",
      "staleAfterHours": 168,
      "query": "$order=created_date DESC"
    }
  ]
}

Interpreting results

The default dataset contains one receipt per checked endpoint. changed means the bounded canonical sample or inferred schema differs from the stored baseline; it does not prove that every row in the source changed. failed describes the observed request or validation failure and retains available response evidence.

This Actor reports source evidence and contract checks. It does not certify a dataset's legal completeness, accuracy, safety, or fitness for a regulated decision.