# CKAN Open Data Portal Scraper (`devilscrapes/ckan-open-data-portal-scraper`) Actor

Query any CKAN open-data portal's Action API -- data.gov.uk, open.canada.ca, data.gov, the EU portal, and national/city catalogues. Give us a portal URL plus package\_search, package\_show, or organization\_list and we page results, retry on throttling, and skip not-found errors so the run keeps going.

- **URL**: https://apify.com/devilscrapes/ckan-open-data-portal-scraper.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<div align="center">
  <img src=".actor/icon.svg" width="160" alt="Devil Scrapes mark" />

## CKAN Open Data Portal Scraper

**💰 $3.05 / 1 000 results**  ·  pay only for results  ·  no credit card to try

*The devil's in the data — we just fetch it clean.* 😈

Query any **CKAN Action API** portal in one run — `data.gov.uk`,
`open.canada.ca`, `data.gov`, the EU open-data portal, and most national and
city catalogues running CKAN. Give us a portal URL plus a search, a dataset
lookup, or an organization roster and we page results, retry on throttling,
and skip logical not-found errors so the rest of your run keeps going.

</div>

***

### 🎯 What this scrapes

**CKAN** (Comprehensive Knowledge Archive Network) is the most widely
deployed open-data portal software in the world, and every deployment
exposes the same standardized **Action API** — `package_search`,
`package_show`, `organization_list` — regardless of which government or
city runs it. `data.gov.uk`, `open.canada.ca`, `data.gov`, the EU portal,
and most national and city open-data catalogues all speak this protocol.

This Actor is a **generic, cross-portal harvester**, not a single-country
scraper. You supply a list of `{portal_url, action}` requests, and we
return every matching record from every request in one run — dataset
metadata, resource listings, or organization rosters, whatever shape that
portal's catalogue happens to publish.

### 🔥 Features — what we handle for you

- 🔁 **We retry with exponential backoff** on `429` and `5xx` responses —
  up to 5 attempts per page, honouring `Retry-After` when a portal sends it.
- 📄 **We page every search for you** — `rows`/`start` walked automatically
  until the result set is exhausted or your `max_items` cap is hit.
- 🧩 **We fan out across portals in one run** — mix a UK search and a
  Canadian search and get valid, disjoint-metadata rows for both, no schema
  conflicts.
- 🧱 **We isolate failures per request** — CKAN's native `success: false`
  "not found" envelope (which arrives on HTTP 200, not an error) is logged
  and skipped; your other requests still complete.
- 🧊 **We keep the dataset clean** — Pydantic-validated rows, ISO-8601
  timestamps, a stable `record_id` when the portal provides one.
- 💰 **You pay only for results that land.** No data → no charge beyond the
  small warm-up fee.

### 💡 Use cases

- **Civic analytics** — pull a national or city open-data catalogue on a
  schedule and track what gets published over time.
- **Journalism / research** — cross-reference multiple government portals
  without writing a client for each country's deployment.
- **Data engineering** — feed dataset metadata into a pipeline or catalogue
  without hand-rolling CKAN Action API calls per portal.
- **Academic / policy research** — bulk-export any CKAN portal's catalogue
  into a typed, analysis-ready format.
- **Compliance / transparency monitoring** — watch an organization's
  published datasets for new entries.

### ⚙️ How to use it

1. Click **Try for free** at the top of the page.
2. Add one or more `requests` — each is `{portal_url, action}` plus optional
   `query` / `filter_query` / `dataset_id` / `rows` / `start` / `max_items` /
   `label`. The prefilled example already searches `data.gov.uk` and
   `open.canada.ca` so you can see multi-portal fan-out on the first run.
3. Click **Start**. Output streams into the run's dataset as each request
   finishes paging.
4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or fetch via
   the API.

### 📥 Input

| Field | Type | Required | Default | Notes |
|---|---|:--:|---|---|
| `requests` | `array` | **yes** | data.gov.uk + open.canada.ca `package_search` prefill | List of `{portal_url, action, query?, filter_query?, dataset_id?, rows?, start?, max_items?, label?}` |
| `proxy_configuration` | `object` | no | `{"useApifyProxy": true}` | Apify Proxy spec |

Each `requests[]` entry:

| Field | Type | Required | Notes |
|---|---|:--:|---|
| `portal_url` | `string` | yes | CKAN portal base URL, e.g. `https://data.gov.uk` |
| `action` | `string` | yes | `package_search`, `package_show`, or `organization_list` |
| `query` | `string` | no | `package_search`'s free-text `q` |
| `filter_query` | `string` | no | `package_search`'s Solr-style `fq`, passed through verbatim |
| `dataset_id` | `string` | required for `package_show` | Dataset slug or UUID |
| `rows` | `integer` | no | `package_search` page size (1–1000), default `100` |
| `start` | `integer` | no | `package_search` paging offset, default `0` |
| `max_items` | `integer` | no | Cap for this request; `0` = unlimited, default `1000` |
| `label` | `string` | no | Echoed into `ResultRow.request_label` |

#### Example input

```json
{
  "requests": [
    {
      "portal_url": "https://data.gov.uk",
      "action": "package_search",
      "rows": 5,
      "label": "data.gov.uk sample search"
    },
    {
      "portal_url": "https://open.canada.ca/data",
      "action": "package_search",
      "rows": 5,
      "label": "open.canada.ca sample search"
    }
  ],
  "proxy_configuration": { "useApifyProxy": true }
}
```

### 📤 Output

Every row is one dataset item. `data` carries the raw CKAN result item
exactly as returned — untyped by design, since a portal's metadata columns
are unknown at build time and change from portal to portal.

| Field | Type | Notes |
|---|---|---|
| `portal_url` | `string` | CKAN portal base URL the row came from |
| `action` | `string` | Which Action API call produced this row |
| `query_used` | `string \| null` | The `q`/`fq`/`id` actually sent, for traceability |
| `request_label` | `string \| null` | Echoed from the request's `label` input field |
| `record_id` | `string \| null` | CKAN `id`/`name` of the dataset or organization, when present |
| `scraped_at` | `string` | ISO-8601 UTC fetch timestamp |
| `data` | `object` | The raw CKAN result item, untouched |

#### Example output

```json
{
  "portal_url": "https://data.gov.uk",
  "action": "package_search",
  "query_used": null,
  "request_label": "data.gov.uk sample search",
  "record_id": "land-registry-price-paid-data",
  "scraped_at": "2026-09-01T09:12:44.000Z",
  "data": {
    "id": "5a67e4a1-0000-0000-0000-000000000000",
    "name": "land-registry-price-paid-data",
    "title": "Price Paid Data",
    "organization": { "name": "land-registry", "title": "HM Land Registry" },
    "num_resources": 4
  }
}
```

### 💰 Pricing

Pay-Per-Event — you pay only when these events fire:

| Event | USD | What it is |
|---|---:|---|
| `actor-start` | $0.05 | One-off warm-up charge per run |
| `result` | $0.003 | Per dataset row emitted |

Example: 1 000 results at the rates above ≈ **$3.05**. No subscription, no
minimum, no card to start — Apify gives every new account free credit.

### 🚧 Limitations

Only the three verified CKAN Action API calls are supported —
`package_search`, `package_show`, `organization_list`. Other CKAN actions
(`resource_search`, `tag_list`, `group_list`, or any write action) are out
of scope. `record_id` is `null` whenever the source item has no `id`/`name`
field, which can happen and is not a bug. This Actor does not normalize
metadata across portals — `data` is the raw result for whatever portal you
point it at, and Socrata portals (a different, unrelated open-data
protocol) are not supported here.

### ❓ FAQ

**Do I need a CKAN account or API key?**

No. Every request in this Actor targets public, unauthenticated CKAN
portals — this Actor runs entirely keyless.

**What happens if I give it a bad `dataset_id`?**

That request is logged and skipped; the run still finishes successfully
and your other requests complete normally. CKAN reports "not found" as a
`success: false` envelope on HTTP 200, and we treat that as a skip, not a
crash.

**What if my `query` or `filter_query` matches nothing?**

The run finishes successfully with zero rows for that request — an empty
result is not an error.

**Can I scrape any CKAN portal, or only the ones in the example?**

Any public CKAN portal — `data.gov.uk` and `open.canada.ca` in the prefill
just demonstrate multi-portal fan-out in a single run.

**Why is `data` a raw untyped object instead of fixed columns?**

Because the metadata shape is unknown at build time — a UK dataset and a
Canadian dataset can carry different extras and licence vocabularies.
Forcing them into one fixed schema would mean dropping real data; `data`
keeps every field the source portal returned.

### 💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an issue
on the Actor's **Issues** tab on Apify Console — we ship fixes weekly and
we read every report.

***

<div align="center">

Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of
opinionated public-data Actors. Honest pricing, real engineering, zero fine print.

</div>

# Actor input Schema

## `requests` (type: `array`):

List of CKAN portal + action requests to run. Each entry: <code>{"portal\_url": "https://data.gov.uk", "action": "package\_search", "query": null, "filter\_query": null, "dataset\_id": null, "rows": 100, "start": 0, "max\_items": 1000, "label": null}</code>. Only <code>portal\_url</code> and <code>action</code> are required per request; <code>dataset\_id</code> is additionally required when <code>action</code> is <code>package\_show</code>.

## `proxy_configuration` (type: `object`):

Apify Proxy spec. CKAN portals are public APIs with no anti-bot wall, but Apify Proxy still absorbs any per-IP throttling under concurrent runs.

## Actor input object example

```json
{
  "requests": [
    {
      "portal_url": "https://data.gov.uk",
      "action": "package_search",
      "rows": 5,
      "max_items": 5,
      "label": "data.gov.uk sample search"
    },
    {
      "portal_url": "https://open.canada.ca/data",
      "action": "package_search",
      "rows": 5,
      "max_items": 5,
      "label": "open.canada.ca sample search"
    }
  ],
  "proxy_configuration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

All dataset items as JSON.

## `datasetItemsCsv` (type: `string`):

Same data exported to CSV.

## `datasetView` (type: `string`):

Open the run dataset in the Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "requests": [
        {
            "portal_url": "https://data.gov.uk",
            "action": "package_search",
            "rows": 5,
            "max_items": 5,
            "label": "data.gov.uk sample search"
        },
        {
            "portal_url": "https://open.canada.ca/data",
            "action": "package_search",
            "rows": 5,
            "max_items": 5,
            "label": "open.canada.ca sample search"
        }
    ],
    "proxy_configuration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/ckan-open-data-portal-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "requests": [
        {
            "portal_url": "https://data.gov.uk",
            "action": "package_search",
            "rows": 5,
            "max_items": 5,
            "label": "data.gov.uk sample search",
        },
        {
            "portal_url": "https://open.canada.ca/data",
            "action": "package_search",
            "rows": 5,
            "max_items": 5,
            "label": "open.canada.ca sample search",
        },
    ],
    "proxy_configuration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/ckan-open-data-portal-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "requests": [
    {
      "portal_url": "https://data.gov.uk",
      "action": "package_search",
      "rows": 5,
      "max_items": 5,
      "label": "data.gov.uk sample search"
    },
    {
      "portal_url": "https://open.canada.ca/data",
      "action": "package_search",
      "rows": 5,
      "max_items": 5,
      "label": "open.canada.ca sample search"
    }
  ],
  "proxy_configuration": {
    "useApifyProxy": true
  }
}' |
apify call devilscrapes/ckan-open-data-portal-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devilscrapes/ckan-open-data-portal-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/yF3nLye0NntnsbB40/builds/JJk6rWzyugusc72Y8/openapi.json