CKAN Open Data Scraper 📂

Search datasets and pull tabular records from any public CKAN open-data portal — data.gov.uk, open.canada.ca and hundreds more. No API key, no proxy.

CKAN is the software behind most government and research open-data portals worldwide. This actor is a universal connector: point it at any CKAN portal and either search its datasets (metadata, organizations, resources, tags) or pull the actual rows from a datastore-backed resource.

Perfect for open-data pipelines, government/research data aggregation, civic tech, data journalism, and keeping a warehouse in sync with public datasets.

Uses CKAN's public Action API — no key, no login. One actor works across every CKAN portal.

---

Two modes

• 🔎 Search datasets — package_search: dataset title, notes, organization, license, tags, and its resources (with datastore_active flags).
• 📊 Resource records — datastore_search: the actual tabular rows from a datastore-backed resource.

Features

• ✅ No API key, no proxy — public CKAN Action API.
• ✅ Universal — works on any CKAN portal (just change the portal URL).
• ✅ Dataset discovery — search + Solr filter queries (res_format:CSV, organization:...).
• ✅ Raw data — pull the rows behind a resource, with full-text search.
• ✅ Auto-pagination — handles both endpoints' paging.

---

Input

Field	Description
Portal URL	Base URL, e.g. https://data.gov.uk or https://open.canada.ca/data.
Mode	datasets (search) or records (datastore rows).
Query	Free-text search.
Filter query	Solr fq filter (datasets mode), e.g. res_format:CSV.
Resource ID	Datastore-active resource ID (records mode).
Max results	Cap on datasets/records returned.

Example — search CSV crime datasets on data.gov.uk

{
  "portalUrl": "https://data.gov.uk",
  "mode": "datasets",
  "query": "crime",
  "filterQuery": "res_format:CSV",
  "maxResults": 500
}

Example — pull rows from a specific resource

{
  "portalUrl": "https://open.canada.ca/data",
  "mode": "records",
  "resourceId": "a811cac0-2a2a-4440-8a81-2994fc753171",
  "maxResults": 10000
}

Output

Datasets mode:

{
  "id": "…",
  "name": "crime-data",
  "title": "Crime",
  "notes": "…",
  "organization": "Calderdale Metropolitan Borough Council",
  "license_title": "UK Open Government Licence (OGL)",
  "num_resources": 3,
  "tags": ["crime", "safety"],
  "url": "https://data.gov.uk/dataset/crime-data",
  "resources": [
    { "id": "…", "name": "2024 CSV", "format": "CSV", "url": "…", "datastore_active": true }
  ]
}

Records mode returns each row as-is (the resource's own columns) plus a _resource_id field.

Workflow tip

Use datasets mode to find resources where datastore_active: true, then feed those resource IDs into records mode to pull the actual data.

Run it on a schedule

Keep a mirror of chosen datasets/resources fresh — pipe to a database, Google Sheets or a webhook.

Notes & limitations

• Public CKAN portals only.
• Records mode requires a datastore-active resource (many resources are just file links, which have no queryable rows).
• Some portals mount CKAN under a subpath (e.g. /data) — include it in the portal URL.
• Data sources: the respective open-data portals; respect their licenses.

---

Keywords

CKAN, open data, data.gov, data.gov.uk, open.canada.ca, government data, datastore, package_search, datasets, civic tech, open data portal, public data, data catalog, research data, data journalism, Solr, open government, data aggregation, tabular data, API.