# Open Data Feed Watcher – Socrata, ArcGIS, CKAN (`loudtzu/open-data-feed-watcher`) Actor

Watch open data datasets and whole portals (Socrata, ArcGIS, CKAN) using metadata only: stale data, column changes, row-count jumps, licence and title changes, endpoint errors, possible personal-info columns. Snapshot kept between runs. JSON/CSV. $0.005 per check.

- **URL**: https://apify.com/loudtzu/open-data-feed-watcher.md
- **Developed by:** [Travis M](https://apify.com/loudtzu) (community)
- **Categories:** Business, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$5.00 / 1,000 check records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Open Data Feed Watcher (Socrata, ArcGIS, CKAN)

**Monitors open data datasets and portals using metadata only** — it never downloads row data — and compares each run with the previous snapshot.

- **Flags:** stale data (vs stated frequency × grace factor), columns added/removed/renamed/retyped, row-count change above a threshold (default 10%), licence and title changes, CKAN resources added/removed, endpoint errors, column names that may hold personal info.
- **Output:** one JSON/CSV record per dataset or portal check: portal, dataset ID, status, HTTP status, age in days, row count and % change, licence, changes, PII-column hints.

**Why not build it yourself?** No per-platform metadata clients, snapshot storage or diff logic to write and maintain.

**Price:** $0.005 per check record (half a cent), no start fee. 20 datasets daily ≈ $0.10/day, about $3/month.

### What it reports, per dataset

- **Stale**: the last update is older than the stated frequency allows (frequency × `staleGraceFactor`, default 1.5). You can set `maxAgeDays` per dataset, which you need for ArcGIS because it states no frequency.
- **Columns** added, removed, renamed or retyped. Renames are matched by Socrata column id, or else by a single removal and addition of the same type (`sameTypeHeuristic`).
- **Row count** change above `rowCountThresholdPct` (default 10%).
- **Licence** and title changes. CKAN resources added or removed.
- **Endpoint errors**: HTTP status or API error, and recovery on the next good run.
- **Possible personal info** (warning only): column *names* such as name, email, phone, address, DOB, owner, SIN, VIN or lat/long. Column contents are never read.

### Whole-portal mode

Under `portals`, give `{"portal": "socrata"|"ckan"|"arcgis", "domain" or "url", "maxDatasets"}`. Each run outputs one `catalogue` record listing the datasets added, removed or retitled since the last run. ArcGIS lists the services under a `/rest/services` URL.

### Metadata endpoints used

| Portal | Calls |
|---|---|
| Socrata | `/api/views/<id>.json`, plus `/api/catalog/v1` for portal scans |
| ArcGIS | layer `?f=json`, `query?where=1=1&returnCountOnly=true` |
| CKAN | `package_show`, `package_search` (portal scan), and `datastore_search?limit=0` (field list and total only, optional) |

### Politeness

It reads `robots.txt` for each host and honours `Crawl-delay`, with a minimum of 1 s between calls to a host. open.canada.ca asks for 20 s; that is also a built-in fallback, because its robots.txt can block plain clients. It sends a normal User-Agent and retries 429 and 5xx responses with backoff.

### Input example

```json
{
  "datasets": [
    {"portal": "socrata", "domain": "data.novascotia.ca", "id": "m6ps-8j6u"},
    {"portal": "arcgis", "url": "https://services2.arcgis.com/11XBiaBYA9Ep0yNJ/arcgis/rest/services/PPLC_Issued_Building_Permits/FeatureServer/0", "maxAgeDays": 14},
    {"portal": "ckan", "domain": "open.canada.ca", "id": "0032ce54-c5dd-4b66-99a0-320a7b5e99f2"}
  ],
  "portals": [{"portal": "socrata", "domain": "data.novascotia.ca", "maxDatasets": 2000}],
  "rowCountThresholdPct": 10,
  "onlyChanges": false
}
```

Use a different `snapshotKey` for each watch list. On the first run every dataset has `firstSeen: true` and no diffs.

### Development

```
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt pytest
.venv/bin/python -m pytest -q
.venv/bin/python -m src     # input in storage/key_value_stores/default/INPUT.json
```

### Pricing

**$0.005 per result** (pay per event, one result = one dataset or portal check record), with no start fee. Watching 20 datasets daily costs about $0.10 a day, roughly $3 a month.

# Actor input Schema

## `datasets` (type: `array`):

Each item: {"portal": "socrata"|"arcgis"|"ckan", "domain": "data.novascotia.ca", "id": "m6ps-8j6u"}. ArcGIS uses "url" = full layer URL (…/FeatureServer/0). CKAN under a path can give "url" (e.g. https://open.canada.ca/data). Optional "maxAgeDays" overrides the stated frequency.

## `portals` (type: `array`):

Each item: {"portal": "socrata"|"ckan"|"arcgis", "domain" or "url", "maxDatasets": 5000}. Reports datasets added, removed or retitled since the last run. ArcGIS url must end in /rest/services. Note: open.canada.ca asks for a 20 s crawl delay, so large CKAN scans are slow.

## `rowCountThresholdPct` (type: `integer`):

Flag when the row count moves more than this percent since the last run.

## `staleGraceFactor` (type: `number`):

A dataset is stale when its age exceeds stated frequency × this factor (>= 1).

## `ckanDatastoreFields` (type: `boolean`):

For CKAN resources in the DataStore, read column definitions and totals (datastore_search limit=0, no rows). Up to 5 resources per dataset.

## `onlyChanges` (type: `boolean`):

Skip unchanged datasets in the output (first sightings are always output).

## `snapshotStore` (type: `string`):

Named store that keeps the previous snapshot between runs.

## `snapshotKey` (type: `string`):

Use different keys to keep separate watch lists.

## Actor input object example

```json
{
  "datasets": [
    {
      "portal": "socrata",
      "domain": "data.novascotia.ca",
      "id": "m6ps-8j6u"
    },
    {
      "portal": "arcgis",
      "url": "https://services2.arcgis.com/11XBiaBYA9Ep0yNJ/arcgis/rest/services/PPLC_Issued_Building_Permits/FeatureServer/0",
      "maxAgeDays": 14
    }
  ],
  "portals": [],
  "rowCountThresholdPct": 10,
  "staleGraceFactor": 1.5,
  "ckanDatastoreFields": true,
  "onlyChanges": false,
  "snapshotStore": "open-data-feed-watcher",
  "snapshotKey": "SNAPSHOT"
}
```

# Actor output Schema

## `checks` (type: `string`):

All dataset and catalogue checks from the run, in the 'overview' table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "datasets": [
        {
            "portal": "socrata",
            "domain": "data.novascotia.ca",
            "id": "m6ps-8j6u"
        },
        {
            "portal": "arcgis",
            "url": "https://services2.arcgis.com/11XBiaBYA9Ep0yNJ/arcgis/rest/services/PPLC_Issued_Building_Permits/FeatureServer/0",
            "maxAgeDays": 14
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("loudtzu/open-data-feed-watcher").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "datasets": [
        {
            "portal": "socrata",
            "domain": "data.novascotia.ca",
            "id": "m6ps-8j6u",
        },
        {
            "portal": "arcgis",
            "url": "https://services2.arcgis.com/11XBiaBYA9Ep0yNJ/arcgis/rest/services/PPLC_Issued_Building_Permits/FeatureServer/0",
            "maxAgeDays": 14,
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("loudtzu/open-data-feed-watcher").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "datasets": [
    {
      "portal": "socrata",
      "domain": "data.novascotia.ca",
      "id": "m6ps-8j6u"
    },
    {
      "portal": "arcgis",
      "url": "https://services2.arcgis.com/11XBiaBYA9Ep0yNJ/arcgis/rest/services/PPLC_Issued_Building_Permits/FeatureServer/0",
      "maxAgeDays": 14
    }
  ]
}' |
apify call loudtzu/open-data-feed-watcher --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,loudtzu/open-data-feed-watcher"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lJ1QnMYfMh7LBMsOs/builds/TLyLTYRewXHkWaBmi/openapi.json