CKAN Open Data Portal Scraper

---

🎯 What this scrapes

CKAN (Comprehensive Knowledge Archive Network) is the most widely deployed open-data portal software in the world, and every deployment exposes the same standardized Action API — package_search, package_show, organization_list — regardless of which government or city runs it. data.gov.uk, open.canada.ca, data.gov, the EU portal, and most national and city open-data catalogues all speak this protocol.

This Actor is a generic, cross-portal harvester, not a single-country scraper. You supply a list of {portal_url, action} requests, and we return every matching record from every request in one run — dataset metadata, resource listings, or organization rosters, whatever shape that portal's catalogue happens to publish.

🔥 Features — what we handle for you

• 🔁 We retry with exponential backoff on 429 and 5xx responses — up to 5 attempts per page, honouring Retry-After when a portal sends it.
• 📄 We page every search for you — rows/start walked automatically until the result set is exhausted or your max_items cap is hit.
• 🧩 We fan out across portals in one run — mix a UK search and a Canadian search and get valid, disjoint-metadata rows for both, no schema conflicts.
• 🧱 We isolate failures per request — CKAN's native success: false "not found" envelope (which arrives on HTTP 200, not an error) is logged and skipped; your other requests still complete.
• 🧊 We keep the dataset clean — Pydantic-validated rows, ISO-8601 timestamps, a stable record_id when the portal provides one.
• 💰 You pay only for results that land. No data → no charge beyond the small warm-up fee.

💡 Use cases

• Civic analytics — pull a national or city open-data catalogue on a schedule and track what gets published over time.
• Journalism / research — cross-reference multiple government portals without writing a client for each country's deployment.
• Data engineering — feed dataset metadata into a pipeline or catalogue without hand-rolling CKAN Action API calls per portal.
• Academic / policy research — bulk-export any CKAN portal's catalogue into a typed, analysis-ready format.
• Compliance / transparency monitoring — watch an organization's published datasets for new entries.

⚙️ How to use it

1. Click Try for free at the top of the page.
2. Add one or more requests — each is {portal_url, action} plus optional query / filter_query / dataset_id / rows / start / max_items / label. The prefilled example already searches data.gov.uk and open.canada.ca so you can see multi-portal fan-out on the first run.
3. Click Start. Output streams into the run's dataset as each request finishes paging.
4. Export from Storage → Dataset as JSON, CSV, or Excel — or fetch via the API.

📥 Input

Field	Type	Required	Default	Notes
requests	array	yes	data.gov.uk + open.canada.ca package_search prefill	List of {portal_url, action, query?, filter_query?, dataset_id?, rows?, start?, max_items?, label?}
proxy_configuration	object	no	{"useApifyProxy": true}	Apify Proxy spec

Each requests[] entry:

Field	Type	Required	Notes
portal_url	string	yes	CKAN portal base URL, e.g. https://data.gov.uk
action	string	yes	package_search, package_show, or organization_list
query	string	no	package_search's free-text q
filter_query	string	no	package_search's Solr-style fq, passed through verbatim
dataset_id	string	required for package_show	Dataset slug or UUID
rows	integer	no	package_search page size (1–1000), default 100
start	integer	no	package_search paging offset, default 0
max_items	integer	no	Cap for this request; 0 = unlimited, default 1000
label	string	no	Echoed into ResultRow.request_label

Example input

{
  "requests": [
    {
      "portal_url": "https://data.gov.uk",
      "action": "package_search",
      "rows": 5,
      "label": "data.gov.uk sample search"
    },
    {
      "portal_url": "https://open.canada.ca/data",
      "action": "package_search",
      "rows": 5,
      "label": "open.canada.ca sample search"
    }
  ],
  "proxy_configuration": { "useApifyProxy": true }
}

📤 Output

Every row is one dataset item. data carries the raw CKAN result item exactly as returned — untyped by design, since a portal's metadata columns are unknown at build time and change from portal to portal.

Field	Type	Notes
portal_url	string	CKAN portal base URL the row came from
action	string	Which Action API call produced this row
query_used	string | null	The q/fq/id actually sent, for traceability
request_label	string | null	Echoed from the request's label input field
record_id	string | null	CKAN id/name of the dataset or organization, when present
scraped_at	string	ISO-8601 UTC fetch timestamp
data	object	The raw CKAN result item, untouched

Example output

{
  "portal_url": "https://data.gov.uk",
  "action": "package_search",
  "query_used": null,
  "request_label": "data.gov.uk sample search",
  "record_id": "land-registry-price-paid-data",
  "scraped_at": "2026-09-01T09:12:44.000Z",
  "data": {
    "id": "5a67e4a1-0000-0000-0000-000000000000",
    "name": "land-registry-price-paid-data",
    "title": "Price Paid Data",
    "organization": { "name": "land-registry", "title": "HM Land Registry" },
    "num_resources": 4
  }
}

💰 Pricing

Pay-Per-Event — you pay only when these events fire:

Event	USD	What it is
actor-start	$0.05	One-off warm-up charge per run
result	$0.003	Per dataset row emitted

Example: 1 000 results at the rates above ≈ $3.05. No subscription, no minimum, no card to start — Apify gives every new account free credit.

🚧 Limitations

Only the three verified CKAN Action API calls are supported — package_search, package_show, organization_list. Other CKAN actions (resource_search, tag_list, group_list, or any write action) are out of scope. record_id is null whenever the source item has no id/name field, which can happen and is not a bug. This Actor does not normalize metadata across portals — data is the raw result for whatever portal you point it at, and Socrata portals (a different, unrelated open-data protocol) are not supported here.

❓ FAQ

Do I need a CKAN account or API key?

No. Every request in this Actor targets public, unauthenticated CKAN portals — this Actor runs entirely keyless.

What happens if I give it a bad dataset_id?

That request is logged and skipped; the run still finishes successfully and your other requests complete normally. CKAN reports "not found" as a success: false envelope on HTTP 200, and we treat that as a skip, not a crash.

What if my query or filter_query matches nothing?

The run finishes successfully with zero rows for that request — an empty result is not an error.

Can I scrape any CKAN portal, or only the ones in the example?

Any public CKAN portal — data.gov.uk and open.canada.ca in the prefill just demonstrate multi-portal fan-out in a single run.

Why is data a raw untyped object instead of fixed columns?

Because the metadata shape is unknown at build time — a UK dataset and a Canadian dataset can carry different extras and licence vocabularies. Forcing them into one fixed schema would mean dropping real data; data keeps every field the source portal returned.

💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an issue on the Actor's Issues tab on Apify Console — we ship fixes weekly and we read every report.

---