# Universal Open Data Portal Fetcher (CKAN + Socrata) (`wdurham/open-data-portal-fetcher`) Actor

Search dataset catalogs and fetch rows from thousands of government open data portals running CKAN or Socrata, via their official APIs. Auto-detects the portal, pages through results, and supports incremental runs.

- **URL**: https://apify.com/wdurham/open-data-portal-fetcher.md
- **Developed by:** [William Durham](https://apify.com/wdurham) (community)
- **Categories:** Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Universal Open Data Portal Fetcher (CKAN + Socrata)

Search dataset catalogs and pull data rows from **thousands of government open data portals** with one Actor. It talks only to the portals' official public APIs (CKAN Action API, CKAN DataStore, Socrata Discovery API, Socrata SODA). No HTML scraping.

- **Auto-detects** whether a portal runs CKAN or Socrata, or lets you force the type
- **Search mode**: one clean metadata item per dataset (title, publisher, tags, license, modified date, resources, URLs)
- **Fetch mode**: the actual rows of a dataset, with paging, a row cap, SoQL (`$select`, `$where`, `$order`, `$q`) for Socrata and DataStore queries/filters for CKAN
- **CKAN files**: if a resource has no DataStore, the CSV / TSV / JSON / GeoJSON / XLSX file is downloaded (with a size cap) and parsed into rows
- **Incremental runs**: remembers what it has seen in a named key-value store and emits only new or changed datasets/rows next time. Ideal for scheduled monitoring (new building permits, new datasets, ...)
- **Robust**: retries with exponential backoff (429/5xx/timeouts, honours `Retry-After`), timeouts, download size caps, clear error messages

### Which portals work?

Any portal running **CKAN** (`/api/3/action/...`) or **Socrata** (`/api/catalog/v1`, `/resource/<id>.json`). Examples tested with this Actor:

| Portal | Platform | Notes |
|---|---|---|
| `https://data.gov.uk` | CKAN | redirects to `ckan.publishing.service.gov.uk`; most resources are file links |
| `https://data.boston.gov` | CKAN | DataStore enabled, fast row paging |
| `https://data.cityofnewyork.us` | Socrata | NYC Open Data |
| `https://data.cityofchicago.org` | Socrata | Chicago Data Portal |

Other well-known CKAN portals include data.gov.au (`https://data.gov.au/data`), open.canada.ca (`https://open.canada.ca/data`) and many EU national portals; other Socrata portals include data.sfgov.org, data.ny.gov, data.wa.gov and data.lacity.org.

**Not supported**: OpenDataSoft, ArcGIS Hub, DKAN, Dataverse or custom catalogs. Note that **catalog.data.gov no longer exposes the CKAN API** (it returns 404 on `/api/3/action/*`), so the Actor reports it as unsupported.

### Input examples

#### Search a Socrata portal for building-permit datasets

```json
{
  "portalUrl": "https://data.cityofchicago.org",
  "mode": "search",
  "query": "building permits",
  "maxItems": 50
}
```

#### Search data.gov.uk (CKAN) with a Solr filter

```json
{
  "portalUrl": "https://data.gov.uk",
  "mode": "search",
  "query": "air quality",
  "ckanFq": "res_format:CSV",
  "maxItems": 200
}
```

#### Fetch Chicago building permits issued since October 2026 (SoQL)

```json
{
  "portalUrl": "https://data.cityofchicago.org",
  "mode": "fetch",
  "datasetId": "ydr8-5enu",
  "soqlWhere": "issue_date >= '2026-10-01'",
  "soqlOrder": "issue_date DESC",
  "maxRows": 5000
}
```

#### Fetch selected columns of NYC DOB Permit Issuance

```json
{
  "portalUrl": "https://data.cityofnewyork.us",
  "mode": "fetch",
  "datasetId": "ipu4-2q9a",
  "soqlSelect": "job__, borough, permit_type, issuance_date",
  "maxRows": 10000,
  "socrataAppToken": "<optional app token>"
}
```

#### Fetch a CKAN DataStore resource (Boston approved building permits)

```json
{
  "portalUrl": "https://data.boston.gov",
  "mode": "fetch",
  "datasetId": "approved-building-permits",
  "filters": { "city": "Charlestown" },
  "maxRows": 2000
}
```

#### Daily monitor: only new/updated Chicago permits

Schedule this input daily. The first run emits everything matching; later runs emit only rows whose Socrata system field `:updated_at` moved past the stored watermark.

```json
{
  "portalUrl": "https://data.cityofchicago.org",
  "mode": "fetch",
  "datasetId": "ydr8-5enu",
  "soqlWhere": "issue_date >= '2026-01-01'",
  "incremental": true,
  "stateStoreName": "chicago-permits-state",
  "maxRows": 50000
}
```

### Input reference (main fields)

| Field | Description |
|---|---|
| `portalUrl` | Portal base URL (any page URL on the portal also works) |
| `portalType` | `auto` (default), `ckan` or `socrata` |
| `mode` | `search` (dataset metadata) or `fetch` (rows) |
| `query`, `maxItems` | Full-text query and cap for search mode |
| `ckanFq`, `ckanSort` | CKAN Solr filter and sort |
| `socrataSearchParams` | Extra Discovery API params, e.g. `{"categories": "Buildings"}` |
| `datasetId` | CKAN package name/id or Socrata 4x4 id |
| `resourceId`, `preferredFormat`, `forceDownload` | Choose which CKAN resource to read and how |
| `soqlSelect`, `soqlWhere`, `soqlOrder`, `soqlQ` | SoQL clauses for Socrata |
| `datastoreQ`, `filters` | CKAN DataStore full-text and exact-match filters (filters are also applied to downloaded files) |
| `maxRows`, `pageSize` | Row cap and rows per request |
| `socrataAppToken` | Optional, secret. Raises Socrata rate limits |
| `incremental`, `incrementalField`, `stateStoreName`, `resetState` | Incremental runs |
| `requestTimeoutSecs`, `maxRetries`, `maxDownloadMb` | Robustness knobs |

### Output

#### Search mode (one item per dataset)

```json
{
  "portal": "https://data.cityofchicago.org",
  "portalType": "socrata",
  "id": "ydr8-5enu",
  "title": "Building Permits",
  "organization": "City of Chicago",
  "tags": ["buildings", "permits"],
  "license": "See Terms of Use",
  "modified": "2026-10-28T12:08:41.000Z",
  "url": "https://data.cityofchicago.org/d/ydr8-5enu",
  "apiUrl": "https://data.cityofchicago.org/resource/ydr8-5enu.json",
  "resources": [{ "id": "ydr8-5enu", "format": "JSON", "url": "https://data.cityofchicago.org/resource/ydr8-5enu.json" }],
  "numResources": 1,
  "changeType": "new"
}
```

`changeType` (`new` / `updated`) is present only in incremental mode. CKAN items list every resource with `format`, `url`, `datastoreActive`, `size` and `lastModified`. The **Datasets** table view in Console shows the key columns.

#### Fetch mode (one item per row)

Rows are passed through unchanged, plus `_portal`, `_datasetId` and (CKAN) `_resourceId`. In Socrata incremental mode the system fields (`:id`, `:updated_at`, ...) are included because they drive the watermark.

### How incremental mode works

State lives in a **named** key-value store (`stateStoreName`, default `open-data-portal-state`), so it survives between runs. One record per query/dataset.

- **Search**: stores each dataset's `modified` value; emits datasets that are new or whose `modified` changed.
- **Socrata fetch**: watermark on `incrementalField` (default `:updated_at`). The query adds `field >= last` and orders by it; rows that share the exact last timestamp and were already emitted are skipped by id, so a `maxRows` cap never loses or duplicates rows between runs.
- **CKAN DataStore fetch**: watermark on `incrementalField` (default `_id`, catches appended rows; use a timestamp column to also catch edits). DataStore cannot filter by range, so the table is scanned and filtered client side.
- **CKAN file resources**: the resource `last_modified` is compared; unchanged file means zero rows, changed file means all rows are emitted again.

Use `resetState: true` to start over.

### Limits and tips

- Socrata without an app token is throttled; add `socrataAppToken` for large pulls.
- CKAN file downloads are capped by `maxDownloadMb` (default 50 MB) and parsed in memory. Legacy `.xls` is not supported.
- Some CKAN portals mark harvested resources as DataStore-active although the DataStore API is blocked; the Actor then falls back to downloading the file.
- Data comes straight from public government portals and may contain names or addresses published by the portal. Process it according to the portal's license and applicable law.

### Cost

The Actor makes plain HTTPS API calls, no browser and no proxy, so compute usage is low: typically well under a minute of 256 MB memory for thousands of rows.

# Actor input Schema

## `portalUrl` (type: `string`):

Base URL of a CKAN or Socrata portal, e.g. https://data.gov.uk, https://data.boston.gov, https://data.cityofnewyork.us, https://data.cityofchicago.org. Any page URL on the portal also works; API/UI suffixes are stripped.

## `portalType` (type: `string`):

Auto-detect by probing the CKAN action API and the Socrata catalog API, or force a type.

## `mode` (type: `string`):

search = one item per dataset (metadata). fetch = data rows of one dataset/resource.

## `query` (type: `string`):

Full-text query for search mode (CKAN q / Socrata Discovery q).

## `maxItems` (type: `integer`):

Maximum number of datasets to return in search mode.

## `ckanFq` (type: `string`):

Solr filter for CKAN package_search, e.g. organization:city-of-boston or res_format:CSV.

## `ckanSort` (type: `string`):

Sort for package_search (default 'metadata_modified desc') or for datastore_search in fetch mode (e.g. '\_id desc').

## `socrataSearchParams` (type: `object`):

Extra query parameters for the Socrata Discovery API, e.g. {"categories": "Buildings"} or {"tags": "permits"}.

## `datasetId` (type: `string`):

fetch mode: CKAN package name/id (e.g. approved-building-permits) or Socrata 4x4 id (e.g. ydr8-5enu).

## `resourceId` (type: `string`):

CKAN only: specific resource id or name. If empty, the first DataStore resource is used, else the first CSV/TSV/JSON/GeoJSON/XLSX file.

## `preferredFormat` (type: `string`):

CKAN only: prefer a resource of this format when no resourceId is given.

## `forceDownload` (type: `boolean`):

CKAN only: download the resource file even if the DataStore API is available.

## `maxRows` (type: `integer`):

Maximum number of rows written in fetch mode.

## `pageSize` (type: `integer`):

Rows per API request (SODA $limit / DataStore limit).

## `soqlSelect` (type: `string`):

Socrata only, e.g. permit\_, issue_date, total_fee.

## `soqlWhere` (type: `string`):

Socrata only, e.g. issue_date > '2026-01-01' AND permit_type = 'PERMIT - NEW CONSTRUCTION'.

## `soqlOrder` (type: `string`):

Socrata only, e.g. issue_date DESC. Defaults to :id for stable paging.

## `soqlQ` (type: `string`):

Socrata only: full-text search within rows.

## `datastoreQ` (type: `string`):

CKAN only: full-text search within DataStore rows.

## `filters` (type: `object`):

CKAN only: {"column": "value"} exact filters (passed to datastore_search, or applied after download for files).

## `socrataAppToken` (type: `string`):

Optional Socrata app token for higher rate limits (sent as X-App-Token).

## `incremental` (type: `boolean`):

Remember what was seen in a named key-value store and only emit new/changed datasets (search) or rows (fetch) on later runs.

## `incrementalField` (type: `string`):

fetch mode: monotonically increasing column. Socrata default ':updated_at' (system field); CKAN DataStore default '\_id'. CKAN file resources use the resource last_modified instead.

## `stateStoreName` (type: `string`):

Named key-value store that keeps incremental state across runs.

## `resetState` (type: `boolean`):

Ignore stored state for this query/dataset and start over (state is then overwritten).

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout.

## `maxRetries` (type: `integer`):

Retries for timeouts, 429 and 5xx (exponential backoff, honours Retry-After).

## `maxDownloadMb` (type: `integer`):

Size cap for CKAN file downloads.

## Actor input object example

```json
{
  "portalUrl": "https://data.cityofchicago.org",
  "portalType": "auto",
  "mode": "search",
  "query": "building permits",
  "maxItems": 100,
  "forceDownload": false,
  "maxRows": 1000,
  "pageSize": 1000,
  "incremental": false,
  "stateStoreName": "open-data-portal-state",
  "resetState": false,
  "requestTimeoutSecs": 60,
  "maxRetries": 4,
  "maxDownloadMb": 50
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset metadata (search mode) or data rows (fetch mode).

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "portalUrl": "https://data.cityofchicago.org",
    "query": "building permits"
};

// Run the Actor and wait for it to finish
const run = await client.actor("wdurham/open-data-portal-fetcher").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "portalUrl": "https://data.cityofchicago.org",
    "query": "building permits",
}

# Run the Actor and wait for it to finish
run = client.actor("wdurham/open-data-portal-fetcher").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "portalUrl": "https://data.cityofchicago.org",
  "query": "building permits"
}' |
apify call wdurham/open-data-portal-fetcher --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,wdurham/open-data-portal-fetcher"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/yd1Wihe0TbbQijQRM/builds/w5fhhgQEoanyXwJQe/openapi.json