# Open Data Scraper — CKAN Government Portals (`hipersoft/ckan-open-data-scraper`) Actor

Search and export datasets from any CKAN open-data portal (data.gov.au, open.canada.ca and many more). Filter by query, organization and file format. One clean row per dataset: title, description, organization, tags, license and every downloadable resource.

- **URL**: https://apify.com/hipersoft/ckan-open-data-scraper.md
- **Developed by:** [hiper soft](https://apify.com/hipersoft) (community)
- **Categories:** Other, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.0005 / dataset scraped

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Open Data Scraper — CKAN Government Portals

Search and export datasets from **any CKAN-based open-data portal** into clean **JSON, CSV, Excel or XML**. CKAN powers hundreds of national, state and city open-data catalogs — including **data.gov.au** (Australia) and **open.canada.ca** (Canada) — and this actor speaks all of them through one uniform interface.

Give it a portal and a search term, and it returns one tidy row per dataset: the **title**, plain-text **description**, publishing **organization**, **tags**, **license** and every downloadable **resource** (file name, format and direct URL) — instead of clicking through the portal one page at a time.

Built for researchers, data journalists, analysts, civic-tech teams and data pipelines that need structured open-government metadata at scale.

### What does the Open Data Scraper do?

Pick a **portal** (a preset like `data.gov.au`, or the base URL of any CKAN catalog), type a **query**, optionally filter by **organization** or **file format**, and the actor paginates through the catalog and returns clean rows:

- One row per **dataset**, with its full metadata and resource list.
- Works across many portals with the same output shape, so you can compare or merge catalogs.
- Optional filters to narrow to a single publisher or to datasets that ship a particular file format (CSV, JSON, GeoJSON, XLSX, and so on).

### What data can you get?

| Field | Type | Description |
| --- | --- | --- |
| `id` | string | Portal's unique dataset identifier |
| `name` | string | Dataset slug (used in its portal URL) |
| `title` | string | Human-readable dataset title |
| `notes` | string | Dataset description, cleaned to plain text |
| `organization` | string | Publishing organization / agency |
| `tags` | array | Keywords / tags attached to the dataset |
| `resources` | array | Downloadable files: `name`, `format`, `url` |
| `numResources` | integer | Number of downloadable resources |
| `license` | string | Dataset license title or identifier |
| `metadataModified` | string | Last time the dataset metadata changed |
| `url` | string | Link to the dataset's page on the portal |

### Supported portals

- **`data.gov.au`** — the Australian Government open-data catalog.
- **`open.canada.ca`** — the Government of Canada open-data catalog.
- **Any other CKAN portal** — type its domain or base URL (for example `catalog.data.gov`, or a state / city catalog) into **Portal** or **Custom base URL**.

> Some portals (for example data.europa.eu) run a different, non-CKAN search interface and are not covered yet. Point the actor at a CKAN catalog, and it will work.

### Use cases

- **Open-data discovery** — find every dataset a government publishes on a topic, with direct download links.
- **Data journalism & research** — build a searchable index of public datasets and their sources.
- **Civic tech & dashboards** — feed clean dataset metadata into your own catalogs, maps or apps.
- **Monitoring** — track when datasets are updated using `metadataModified`.
- **Data engineering** — enumerate CSV/GeoJSON resources across a catalog to drive downstream ingestion.

### How to use it

1. Add the Open Data Scraper to your Apify account and open the input form.
2. Choose a **Portal** — a preset such as `data.gov.au`, or paste any CKAN portal's base URL.
3. Enter a **Search query** (e.g. `health`), or leave it empty to list everything.
4. Optionally filter by **Organization** slug or **Resource format** (e.g. `CSV`).
5. Set **Max datasets** to cap the export, then **Run**.
6. **Export** the dataset as JSON, CSV, Excel or XML, or pull it via the Apify API.

### Input example

```json
{
  "portal": "data.gov.au",
  "query": "health",
  "format": "CSV",
  "maxItems": 100
}
```

| Field | Type | Description |
| --- | --- | --- |
| `portal` | string | Preset key (`data.gov.au`, `open.canada.ca`) or a CKAN portal domain / base URL. |
| `baseUrl` | string | Optional custom CKAN base URL, used instead of `portal` when set. |
| `query` | string | Free-text search term. Empty lists all datasets. |
| `organization` | string | Optional organization slug filter. |
| `format` | string | Optional resource file-format filter (e.g. `CSV`). |
| `maxItems` | integer | Maximum number of datasets to return. |

### Output example

```json
{
  "id": "a9372b1f-e5c0-4d03-9714-0dc707343f2f",
  "name": "mental-health-data",
  "title": "Mental Health Data",
  "notes": "Time series state level datasets showing important indicators regarding mental health.",
  "organization": "SA Health",
  "tags": ["mental health"],
  "resources": [
    { "name": "Readmissions", "format": "CSV", "url": "https://data.gov.au/.../readmissions.csv" }
  ],
  "numResources": 3,
  "license": "Creative Commons Attribution 3.0 Australia",
  "metadataModified": "2025-06-25T14:55:56.217841",
  "url": "https://data.gov.au/data/dataset/mental-health-data"
}
```

### FAQ

**What is CKAN?**
CKAN is the open-source data-management system that runs most government open-data portals worldwide. Because every CKAN portal shares the same catalog structure, one actor can search across many of them and return a consistent output.

**Which portals work?**
`data.gov.au` and `open.canada.ca` are built in as presets, and you can point the actor at any other CKAN catalog by entering its domain or base URL.

**How many datasets can I get?**
As many as your search returns. Use **Max datasets** to cap a run and keep it cheap.

**How does billing work?**
You pay per dataset you get, so small, tightly filtered runs stay inexpensive.

**What export formats are supported?**
JSON, CSV, Excel and XML, plus direct access via the Apify API.

**Can I use this with n8n?**
Yes. Run the Open Data Scraper from n8n with the [Apify node](https://n8n.io/integrations/apify/) — trigger a run, wait for it to finish, and read the dataset items straight into your workflow to build catalogs, enrich records or drive alerts. It also works with [Make](https://apify.com/integrations/make), [Zapier](https://apify.com/integrations/zapier) and the [Apify API](https://docs.apify.com/api/v2).

**Can I connect this to other tools?**
The Open Data Scraper connects with almost any cloud service or web app through [integrations on the Apify platform](https://apify.com/integrations) — [Make](https://apify.com/integrations/make), [Zapier](https://apify.com/integrations/zapier), [Slack](https://docs.apify.com/platform/integrations/slack), [Google Drive](https://docs.apify.com/platform/integrations/drive) and [many more](https://apify.com/integrations), plus [webhooks](https://docs.apify.com/platform/integrations/webhooks) to trigger an action whenever a run finishes.

**Is this only public data?**
Yes. It returns only openly published open-government dataset metadata. You are responsible for compliance with each portal's terms and the licence of each dataset.

### Notes

Original clean-room implementation. Not affiliated with any government or data portal. Dataset metadata is published by the respective portals under their own open licences; cite the original source where required.

# Actor input Schema

## `portal` (type: `string`):

Which open-data portal to search. Pick a preset (data.gov.au = Australia, open.canada.ca = Canada) or type any CKAN portal's base URL / domain, e.g. "catalog.data.gov" or "https://data.overheid.nl". Leave the Base URL field empty to use this value.

## `baseUrl` (type: `string`):

Optional. The base URL of a CKAN portal, used instead of Portal when set. This is the address the API sits on, e.g. "https://open.canada.ca/data/en" or "https://catalog.data.gov". You can even paste a full package\_search URL; the actor trims it back to the base automatically.

## `query` (type: `string`):

Free-text search across dataset titles, descriptions and tags, e.g. "health", "transport budget" or "air quality". Leave empty to list all datasets on the portal (newest matches first).

## `organization` (type: `string`):

Optional. Restrict results to a single publishing organization, using its CKAN organization slug (the value seen in the dataset's organization URL), e.g. "sa-health-south-australia-government".

## `format` (type: `string`):

Optional. Only return datasets that contain at least one resource in this file format, e.g. "CSV", "JSON", "GeoJSON", "XLSX" or "PDF".

## `maxItems` (type: `integer`):

Maximum number of datasets to return. Keep it low to keep runs fast and cheap; raise it to pull a whole catalog.

## Actor input object example

```json
{
  "portal": "data.gov.au",
  "baseUrl": "https://catalog.data.gov",
  "query": "health",
  "organization": "sa-health-south-australia-government",
  "format": "CSV",
  "maxItems": 100
}
```

# Actor output Schema

## `results` (type: `string`):

The results as dataset items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "portal": "data.gov.au",
    "query": "health",
    "maxItems": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("hipersoft/ckan-open-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "portal": "data.gov.au",
    "query": "health",
    "maxItems": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("hipersoft/ckan-open-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "portal": "data.gov.au",
  "query": "health",
  "maxItems": 100
}' |
apify call hipersoft/ckan-open-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hipersoft/ckan-open-data-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wL1YEuVgJcXij33FZ/builds/eXiQTo4nFOhoMQZSi/openapi.json