Universal Open Data Portal Fetcher (CKAN + Socrata)

Search dataset catalogs and pull data rows from thousands of government open data portals with one Actor. It talks only to the portals' official public APIs (CKAN Action API, CKAN DataStore, Socrata Discovery API, Socrata SODA). No HTML scraping.

• Auto-detects whether a portal runs CKAN or Socrata, or lets you force the type
• Search mode: one clean metadata item per dataset (title, publisher, tags, license, modified date, resources, URLs)
• Fetch mode: the actual rows of a dataset, with paging, a row cap, SoQL ($select, $where, $order, $q) for Socrata and DataStore queries/filters for CKAN
• CKAN files: if a resource has no DataStore, the CSV / TSV / JSON / GeoJSON / XLSX file is downloaded (with a size cap) and parsed into rows
• Incremental runs: remembers what it has seen in a named key-value store and emits only new or changed datasets/rows next time. Ideal for scheduled monitoring (new building permits, new datasets, ...)
• Robust: retries with exponential backoff (429/5xx/timeouts, honours Retry-After), timeouts, download size caps, clear error messages

Which portals work?

Any portal running CKAN (/api/3/action/...) or Socrata (/api/catalog/v1, /resource/<id>.json). Examples tested with this Actor:

Portal	Platform	Notes
https://data.gov.uk	CKAN	redirects to ckan.publishing.service.gov.uk; most resources are file links
https://data.boston.gov	CKAN	DataStore enabled, fast row paging
https://data.cityofnewyork.us	Socrata	NYC Open Data
https://data.cityofchicago.org	Socrata	Chicago Data Portal

Other well-known CKAN portals include data.gov.au (https://data.gov.au/data), open.canada.ca (https://open.canada.ca/data) and many EU national portals; other Socrata portals include data.sfgov.org, data.ny.gov, data.wa.gov and data.lacity.org.

Not supported: OpenDataSoft, ArcGIS Hub, DKAN, Dataverse or custom catalogs. Note that catalog.data.gov no longer exposes the CKAN API (it returns 404 on /api/3/action/*), so the Actor reports it as unsupported.

Input examples

Search a Socrata portal for building-permit datasets

{
  "portalUrl": "https://data.cityofchicago.org",
  "mode": "search",
  "query": "building permits",
  "maxItems": 50
}

Search data.gov.uk (CKAN) with a Solr filter

{
  "portalUrl": "https://data.gov.uk",
  "mode": "search",
  "query": "air quality",
  "ckanFq": "res_format:CSV",
  "maxItems": 200
}

Fetch Chicago building permits issued since October 2026 (SoQL)

{
  "portalUrl": "https://data.cityofchicago.org",
  "mode": "fetch",
  "datasetId": "ydr8-5enu",
  "soqlWhere": "issue_date >= '2026-10-01'",
  "soqlOrder": "issue_date DESC",
  "maxRows": 5000
}

Fetch selected columns of NYC DOB Permit Issuance

{
  "portalUrl": "https://data.cityofnewyork.us",
  "mode": "fetch",
  "datasetId": "ipu4-2q9a",
  "soqlSelect": "job__, borough, permit_type, issuance_date",
  "maxRows": 10000,
  "socrataAppToken": "<optional app token>"
}

Fetch a CKAN DataStore resource (Boston approved building permits)

{
  "portalUrl": "https://data.boston.gov",
  "mode": "fetch",
  "datasetId": "approved-building-permits",
  "filters": { "city": "Charlestown" },
  "maxRows": 2000
}

Daily monitor: only new/updated Chicago permits

Schedule this input daily. The first run emits everything matching; later runs emit only rows whose Socrata system field :updated_at moved past the stored watermark.

{
  "portalUrl": "https://data.cityofchicago.org",
  "mode": "fetch",
  "datasetId": "ydr8-5enu",
  "soqlWhere": "issue_date >= '2026-01-01'",
  "incremental": true,
  "stateStoreName": "chicago-permits-state",
  "maxRows": 50000
}

Input reference (main fields)

Field	Description
portalUrl	Portal base URL (any page URL on the portal also works)
portalType	auto (default), ckan or socrata
mode	search (dataset metadata) or fetch (rows)
query, maxItems	Full-text query and cap for search mode
ckanFq, ckanSort	CKAN Solr filter and sort
socrataSearchParams	Extra Discovery API params, e.g. {"categories": "Buildings"}
datasetId	CKAN package name/id or Socrata 4x4 id
resourceId, preferredFormat, forceDownload	Choose which CKAN resource to read and how
soqlSelect, soqlWhere, soqlOrder, soqlQ	SoQL clauses for Socrata
datastoreQ, filters	CKAN DataStore full-text and exact-match filters (filters are also applied to downloaded files)
maxRows, pageSize	Row cap and rows per request
socrataAppToken	Optional, secret. Raises Socrata rate limits
incremental, incrementalField, stateStoreName, resetState	Incremental runs
requestTimeoutSecs, maxRetries, maxDownloadMb	Robustness knobs

Output

Search mode (one item per dataset)

{
  "portal": "https://data.cityofchicago.org",
  "portalType": "socrata",
  "id": "ydr8-5enu",
  "title": "Building Permits",
  "organization": "City of Chicago",
  "tags": ["buildings", "permits"],
  "license": "See Terms of Use",
  "modified": "2026-10-28T12:08:41.000Z",
  "url": "https://data.cityofchicago.org/d/ydr8-5enu",
  "apiUrl": "https://data.cityofchicago.org/resource/ydr8-5enu.json",
  "resources": [{ "id": "ydr8-5enu", "format": "JSON", "url": "https://data.cityofchicago.org/resource/ydr8-5enu.json" }],
  "numResources": 1,
  "changeType": "new"
}

changeType (new / updated) is present only in incremental mode. CKAN items list every resource with format, url, datastoreActive, size and lastModified. The Datasets table view in Console shows the key columns.

Fetch mode (one item per row)

Rows are passed through unchanged, plus _portal, _datasetId and (CKAN) _resourceId. In Socrata incremental mode the system fields (:id, :updated_at, ...) are included because they drive the watermark.

How incremental mode works

State lives in a named key-value store (stateStoreName, default open-data-portal-state), so it survives between runs. One record per query/dataset.

• Search: stores each dataset's modified value; emits datasets that are new or whose modified changed.
• Socrata fetch: watermark on incrementalField (default :updated_at). The query adds field >= last and orders by it; rows that share the exact last timestamp and were already emitted are skipped by id, so a maxRows cap never loses or duplicates rows between runs.
• CKAN DataStore fetch: watermark on incrementalField (default _id, catches appended rows; use a timestamp column to also catch edits). DataStore cannot filter by range, so the table is scanned and filtered client side.
• CKAN file resources: the resource last_modified is compared; unchanged file means zero rows, changed file means all rows are emitted again.

Use resetState: true to start over.

Limits and tips

• Socrata without an app token is throttled; add socrataAppToken for large pulls.
• CKAN file downloads are capped by maxDownloadMb (default 50 MB) and parsed in memory. Legacy .xls is not supported.
• Some CKAN portals mark harvested resources as DataStore-active although the DataStore API is blocked; the Actor then falls back to downloading the file.
• Data comes straight from public government portals and may contain names or addresses published by the portal. Process it according to the portal's license and applicable law.

Cost

The Actor makes plain HTTPS API calls, no browser and no proxy, so compute usage is low: typically well under a minute of 256 MB memory for thousands of rows.