# CKAN Open Data Scraper - Datasets & Resources (`logiover/ckan-open-data-portal-scraper`) Actor

Scrape any CKAN open-data portal in bulk. Extract dataset title, description, publishing organisation, licence, tags, resource formats and download URLs, plus created and modified dates to CSV/JSON. No API key, no login.

- **URL**: https://apify.com/logiover/ckan-open-data-portal-scraper.md
- **Developed by:** [Logiover](https://apify.com/logiover) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## CKAN Open Data Scraper - Datasets & Resources

![Apify Actor](https://img.shields.io/badge/Apify-Actor-00A67E?logo=apify\&logoColor=white) ![No API key](https://img.shields.io/badge/No%20API%20key-required-2ea44f) ![Pay per result](https://img.shields.io/badge/Pricing-Pay%20per%20result-1C7ED6) ![Export](https://img.shields.io/badge/Export-JSON%20%7C%20CSV%20%7C%20Excel-F59E0B)

**Scrape any CKAN open-data portal in bulk. Extract dataset title, description, publishing organisation, licence, tags, resource formats and download URLs, plus created and modified dates to CSV/JSON. No API key, no login.**

***

### What does the CKAN Scraper do?

This Actor turns CKAN data into a structured dataset. You point it at whole catalogues or filtered by topic, it walks the result pages one after another, and it writes one clean row per dataset into your dataset — ready to export as JSON, CSV or Excel, or to pull straight from the Apify API.

CKAN is the software behind most national and municipal open-data portals, and every installation exposes the same keyless `package_search` API. The Actor reads that endpoint, so one configuration works against data.gov.uk, the EU portal, open.canada.ca and hundreds of city portals. Resource formats and download URLs are flattened onto the dataset row, which turns the export into a working download manifest. Pagination is followed automatically until it runs out of results, hits your page limit or hits your **Max items** cap, whichever comes first. Every row is de-duplicated across the whole run, so you are never billed twice for the same dataset.

There is **no API key, no login and no browser** involved. That keeps runs fast and cheap, and it means you can schedule the Actor without worrying about credentials expiring.

### Who is it for?

- **Data engineers** locating public datasets to ingest.
- **Researchers and analysts** surveying what a government publishes.
- **AI teams** sourcing openly licensed data with clear provenance.
- **Journalists** finding records for an investigation.
- **Civic-tech developers** building tools on public data.

### Use cases

- Export every dataset a portal publishes with its download links.
- Filter to openly licensed datasets you can legally reuse.
- See which organisations publish the most, and how recently.
- Find datasets available in a specific format such as CSV or GeoJSON.
- Monitor a portal for newly added datasets on a schedule.

### Why use this CKAN Scraper?

- 🔑 **Keyless** — no account, no API token, no cookies to paste.
- 📦 **22 fields per dataset** — everything the source exposes, already typed.
- 📄 **Real pagination** — it walks page after page instead of returning the first screen.
- 🎯 **Precise caps** — **Max items** stops the run exactly where you want it, so the bill is predictable.
- 📊 **Export anywhere** — JSON, CSV, Excel or HTML, plus the Apify API and integrations.
- 💸 **Pay per result** — you pay for rows you actually receive, with no platform fees to calculate.

### What data can you extract?

Every run produces one row per dataset, with these fields:

| Field | Type | Description |
|-------|------|-------------|
| `portalDomain` | string | Domain of the CKAN portal this row came from |
| `datasetId` | string | CKAN dataset identifier |
| `name` | string | URL name of the dataset |
| `title` | string | Dataset title |
| `datasetUrl` | string | Link to the dataset page on the portal |
| `notes` | string | Dataset description with HTML stripped |
| `organization` | string | Publishing organisation name |
| `organizationTitle` | string | Publishing organisation display title |
| `licenseId` | string | Licence identifier |
| `licenseTitle` | string | Licence name |
| `isOpen` | boolean | Whether CKAN considers the licence open |
| `tags` | string | Tags, separated by a pipe |
| `groups` | string | Groups or themes, separated by a pipe |
| `resourceCount` | number | Number of downloadable resources |
| `resourceFormats` | string | Distinct resource formats, separated by a pipe |
| `resourceUrls` | string | Resource download URLs, separated by a pipe |
| `state` | string | CKAN record state, normally active |
| `createdAt` | string | ISO timestamp the metadata was created |
| `modifiedAt` | string | ISO timestamp the metadata was last modified |
| `query` | string | Search query this row came from |
| `page` | number | Result page the dataset appeared on |
| `scrapedAt` | string | ISO timestamp of extraction |

#### Output example

```json
{
  "portalDomain": "data.gov.uk",
  "datasetId": "e9fe97f5-bb00-44e8-a06a-8fab0ef18800",
  "name": "surface-water-flood-maps-frm-fh-surface-water-depth-m536",
  "title": "Surface Water Flood Maps (FRM_FH_SURFACE_WATER_DEPTH_M)",
  "datasetUrl": "https://data.gov.uk/dataset/surface-water-flood-maps-frm-fh-surface-water-depth-m536",
  "notes": "The following probabilities are available for surface water flooding: - High - 10 year return period - Medium - 200 year return period - Low - 200 year return period plus climate\u2026",
  "organization": "scottish-government-spatial-data-infrastructure",
  "organizationTitle": "Scottish Government SpatialData.gov.scot",
  "licenseId": null,
  "licenseTitle": null,
  "isOpen": false,
  "tags": null,
  "groups": null,
  "resourceCount": 2,
  "resourceFormats": "ZIP",
  "resourceUrls": "https://www.sepa.org.uk/environment/environmental-data/ | https://www2.sepa.org.uk/flooddata/SEPA_Surface_Water_Flood_Maps_v2_1.zip",
  "state": "active",
  "createdAt": "2026-09-18T17:06:16.481721",
  "modifiedAt": "2026-09-18T17:06:16.481725",
  "query": "whole catalogue",
  "page": 1,
  "scrapedAt": "2026-09-19T11:22:03.517Z"
}
```

### How to use

#### Option A — whole catalogues

```json
{
  "maxItems": 2000,
  "maxPagesPerPortal": 10,
  "portalUrls": [
    "https://data.gov.uk",
    "https://open.canada.ca/data"
  ]
}
```

1. Open the Actor and fill in the first field.
2. Set the page limit and **Max items** to bound the run.
3. Click **Start**, then export from the **Output** tab.

#### Option B — filtered by topic

```json
{
  "maxItems": 2000,
  "maxPagesPerPortal": 10,
  "portalUrls": [
    "https://data.gov.uk"
  ],
  "searchQueries": [
    "transport",
    "housing"
  ]
}
```

Everything in the first field is processed independently, so you can batch several targets into one run and split them apart afterwards.

### Input parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `portalUrls` | array | `–` | Home pages of the CKAN open-data portals to scrape, e. |
| `searchQueries` | array | `[]` | Terms to search for on each portal. |
| `maxPagesPerPortal` | integer | `4` | How many result pages to walk for each portal. |
| `maxItems` | integer | `500` | Stop after this many datasets. |
| `maxConcurrency` | integer | `2` | Parallel requests. |
| `proxyConfiguration` | object | `{"useApifyProxy": true}` | Proxy used to fetch pages. |

### Tips for best results

- Paste the portal's home page — the Actor appends the CKAN API path for you.
- Leave the query empty to walk the whole portal; add one to focus the run.
- The API accepts up to 1,000 datasets per request, so exports need very few calls.
- `resourceUrls` gives you a ready download manifest without opening a single dataset page.
- `isOpen` is the quickest filter when you intend to redistribute the data.
- `organization` is the best way to slice a national portal into publishers.
- Some portals mount CKAN under a path such as `/data`; paste that full path and it still works.
- Schedule a monthly run to track what a portal newly published.
- Run several portals in one job and split them afterwards with `portalDomain`.
- CKAN defines author, maintainer and version fields, but three national portals filled none of them across 90 sampled datasets, so they are not included.
- Licence and group columns are filled by some portals and left blank by others — data.gov.uk publishes no licence, open.canada.ca publishes one for every dataset.

### Integrations

Send results straight into the tools you already use: **Google Sheets**, **Slack**, **Zapier**, **Make**, **Airtable** or any **Webhook**. You can also **schedule** the Actor to run hourly, daily or weekly and have each run append to the same dataset, which is how you build a price or availability history rather than a one-off snapshot.

### API usage

Run the Actor and collect results from any language. Replace `<YOUR_TOKEN>` with your Apify API token.

**cURL**

```bash
curl -X POST "https://api.apify.com/v2/acts/logiover~ckan-open-data-portal-scraper/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"maxItems": 2000, "maxPagesPerPortal": 10, "portalUrls": ["https://data.gov.uk", "https://open.canada.ca/data"]}'
```

**Node.js**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_TOKEN>' });
const run = await client.actor('logiover/ckan-open-data-portal-scraper').call({"maxItems": 2000, "maxPagesPerPortal": 10, "portalUrls": ["https://data.gov.uk", "https://open.canada.ca/data"]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient('<YOUR_TOKEN>')
run = client.actor('logiover/ckan-open-data-portal-scraper').call(run_input={"maxItems": 2000, "maxPagesPerPortal": 10, "portalUrls": ["https://data.gov.uk", "https://open.canada.ca/data"]})
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)
```

### Use with AI agents (MCP)

This Actor is available through the Apify MCP server, so an AI agent can call it as a tool. Point your agent at `https://mcp.apify.com` and it can run the CKAN Scraper on demand — for example: *"Pull the first 500 datasets from CKAN and summarise what you find."* The agent receives the same structured rows you would get from the UI.

### FAQ

#### Do I need a CKAN account or API key?

No. The Actor reads publicly available data only. There is nothing to authenticate and no credentials to rotate.

#### How many datasets can I get in one run?

As many as the source exposes. Raise the page limit and **Max items** together; the run stops at whichever limit it reaches first.

#### Why did I get fewer rows than I asked for?

The source ran out of datasets. That is normal for narrow searches — broaden the query or add more targets to one run.

#### Are results de-duplicated?

Yes. Each dataset is emitted once per run, even when it appears on several pages, so you are never billed twice for the same record.

#### Why are some fields empty?

CKAN does not publish every attribute for every dataset. Empty means the source did not supply it, not that extraction failed.

#### What export formats are supported?

JSON, CSV, Excel, HTML and RSS from the **Output** tab, plus the Apify API and any integration you connect.

#### How fast is it?

It is pure HTTP with no browser, so a page of results typically takes a second or two. Raise **Max concurrency** carefully — the source rate-limits aggressive crawling.

#### Can I schedule it?

Yes. Use the Apify scheduler to run it on any interval and append each run to the same dataset for time-series analysis.

#### Does it work behind a proxy?

It uses Apify Proxy automatically. You can switch groups or supply your own proxies in **Proxy configuration**.

#### How often does the data change?

CKAN updates continuously. Re-run whenever you need current data; the Actor always reads the live source, never a cache.

#### Is the output schema stable?

Yes. Field names and types are fixed, so downstream pipelines will not break between runs.

#### What if the source changes its format?

Open an issue on the **Issues** tab and it gets fixed. The Actor is actively maintained.

### Is it legal?

This Actor reads only publicly available data from CKAN — the same content any visitor or client sees without logging in. It does not bypass authentication, does not collect private data and does not attempt to defeat access controls. You are responsible for how you use the output: respect the source's terms of service, applicable copyright, and data-protection law such as GDPR where personal data is involved. Scraping public data is generally lawful in the EU and the US, but the responsibility for the downstream use of that data sits with you.

### Related scrapers

- [Eurostat Scraper](https://apify.com/logiover/eurostat-data-scraper) — European statistics
- [World Bank Scraper](https://apify.com/logiover/world-bank-data-scraper) — Global indicators
- [Zenodo Scraper](https://apify.com/logiover/zenodo-scraper) — Research datasets
- [DataCite Scraper](https://apify.com/logiover/datacite-doi-scraper) — Dataset DOIs

# Changelog

This Actor's version history is a separate document: https://apify.com/logiover/ckan-open-data-portal-scraper/changelog.md

# Actor input Schema

## `portalUrls` (type: `array`):

Home pages of the CKAN open-data portals to scrape, e.g. https://data.gov.uk. The Actor appends the CKAN API path itself.

## `searchQueries` (type: `array`):

Terms to search for on each portal. Leave empty to walk the whole catalogue.

## `maxPagesPerPortal` (type: `integer`):

How many result pages to walk for each portal. Set 0 for no limit.

## `maxItems` (type: `integer`):

Stop after this many datasets. Set 0 for no limit.

## `maxConcurrency` (type: `integer`):

Parallel requests. Lower this if the source rate-limits you.

## `proxyConfiguration` (type: `object`):

Proxy used to fetch pages. Defaults to Apify Proxy (automatic). You usually do not need to change this.

## Actor input object example

```json
{
  "portalUrls": [
    "https://data.gov.uk",
    "https://open.canada.ca/data"
  ],
  "searchQueries": [],
  "maxPagesPerPortal": 4,
  "maxItems": 500,
  "maxConcurrency": 2,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All datasets extracted in this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "portalUrls": [
        "https://data.gov.uk",
        "https://open.canada.ca/data"
    ],
    "maxPagesPerPortal": 4,
    "maxItems": 500,
    "maxConcurrency": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("logiover/ckan-open-data-portal-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "portalUrls": [
        "https://data.gov.uk",
        "https://open.canada.ca/data",
    ],
    "maxPagesPerPortal": 4,
    "maxItems": 500,
    "maxConcurrency": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("logiover/ckan-open-data-portal-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "portalUrls": [
    "https://data.gov.uk",
    "https://open.canada.ca/data"
  ],
  "maxPagesPerPortal": 4,
  "maxItems": 500,
  "maxConcurrency": 2
}' |
apify call logiover/ckan-open-data-portal-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,logiover/ckan-open-data-portal-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dsCeQIBpdZuO2Hq6O/builds/ZVS6C5enICVctRUYW/openapi.json