# Data Gov Uk Dataset Scraper (`b2b_lead_generation/data-gov-uk-dataset-scraper`) Actor

Easily search and extract dataset metadata or downloadable file links from the UK Open Data catalog. Gather dataset details, file formats, and licensing information, then export your results instantly as JSON, CSV, or Excel.

- **URL**: https://apify.com/b2b_lead_generation/data-gov-uk-dataset-scraper.md
- **Developed by:** [B2B Lead Generation](https://apify.com/b2b_lead_generation) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## data.gov.uk Dataset Scraper: UK Open Data

<p align="center">
  <img src="https://imgur.com/Rn3Nz4t.png" alt="data.gov.uk Dataset" style="max-width:100%; border-radius:14px; width:680px;" />
</p>

Search the official UK open data catalogue to extract comprehensive dataset records or attached resource files. Gather structured metadata—including dataset titles, publisher details, resource formats, and licence types—into clean datasets ready for instant export to JSON, CSV, or Excel.

Instead of manually browsing through catalogue pages, you can collect UK public sector data records in bulk for research, compliance, and analysis.

### Use Cases

- **Catalogue research:** Search UK open data for specific topics (such as health or transport) and maintain structured metadata in a single table.
- **Resource & file inventory:** Switch to resources mode to list downloadable files (CSV, JSON, PDF) linked across datasets.
- **Publisher filtering:** Apply search filters to target specific government departments or organizations.
- **Automated updates:** Run scheduled queries to track changes and updates across the UK open data ecosystem.
- **Licence compliance:** Check dataset licensing details (`license_title`, `license_url`) before reusing attached files.

### Extracted Data Points

#### Search Mode (Dataset Records)

- Dataset ID, title, and package type
- Description, notes, and publishing organization
- Primary and secondary themes
- Creation and modification timestamps
- Licence title, ID, and licence URL
- Attached resources list (when enabled)

#### Resources Mode (File Records)

- Resource ID, direct download URL, and file format (e.g., CSV, PDF, GeoJSON)
- Parent dataset ID, title, and publisher metadata

### Input Parameters

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `mode` | String | `search` | `search` exports one row per dataset. `resources` exports one row per attached file. |
| `query` | String | `health` | Search terms for the catalogue query. |
| `filterQuery` | String | empty | Optional filter query (e.g., `organization:cabinet-office`). |
| `includeResources` | Boolean | `true` | In search mode, includes the attached file list within each dataset record. |
| `maxItems` | Integer | `100` | Maximum number of rows to retrieve (up to 1,000). |
| `timeoutSecs` | Integer | `300` | Overall execution limit in seconds. |

#### Example Input

```json
{
    "mode": "search",
    "query": "health",
    "filterQuery": "",
    "includeResources": true,
    "maxItems": 100,
    "timeoutSecs": 300
}

```

### Output Format

```json
{
    "id": "d44b4f65-563a-4e3c-9a01-f51b4a4dbdd2",
    "isopen": true,
    "license_id": "uk-ogl",
    "license_title": "UK Open Government Licence (OGL)",
    "license_url": "https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/",
    "metadata_created": "2016-06-10T09:56:02.268810",
    "metadata_modified": "2026-07-08T13:33:16.922355",
    "name": "health",
    "notes": "Reference Costs provide indicators of the unit costs of providing a range of hospital and community services at trust and hospital level.",
    "organization": {
        "id": "8549022f-fa16-43e4-a56f-6a45ca875199",
        "name": "open-data-ni",
        "title": "OpenDataNI"
    },
    "theme-primary": "health",
    "datasetTitle": "Health Trust Reference Costs 2014-15",
    "datasetType": "dataset"
}

```

### How It Works

1. Processes configuration inputs and sets search parameters or organization filters.
2. Pages through catalogue results efficiently while automatically skipping duplicate record IDs.
3. Manages temporary connection disruptions smoothly with retry delays.
4. Logs invalid query filters as error entries while allowing remaining queries to run smoothly.
5. Generates clean datasets available for download in JSON, CSV, or Excel formats.

### Frequently Asked Questions

**What is the difference between search mode and resources mode?**

`search` mode outputs one record per dataset (including dataset metadata). `resources` mode outputs one record per individual file attached to those datasets, featuring direct file URLs and format labels.

**What happens if a filter query is rejected or invalid?**

An error entry explaining the issue is logged, while the process continues cleanly without crashing.

**Can I export datasets directly to CSV or Excel?**

Yes. Completed datasets can be downloaded instantly in JSON, CSV, or Excel formats from your run overview.

***

### Legal & Compliance

#### Trademark Notice

data.gov.uk Dataset and related names, logos, and trademarks belong to their respective owners. Their mention in this documentation is strictly to identify data utility and compatibility, and does not imply affiliation, sponsorship, or endorsement.

#### Disclaimer

This tool is an independent utility and is not affiliated with, endorsed by, or officially connected to third-party public catalogue providers. Users are responsible for ensuring their usage complies with relevant terms of service, licence conditions, and guidelines.

#### Is web scraping legal?

This tool collects publicly available catalogue records without bypassing access controls. Collecting public data is standard practice, but proper compliance requires considering how the data is used:

- **Personal data:** Ensure dataset usage respects regional privacy regulations (such as GDPR or CCPA) when combining outputs with personal user records.
- **Copyright & Terms:** Respect the terms of use, usage limits, and licensing terms (such as the UK Open Government Licence) set by data sources.
- **Legal advice:** This summary is provided for general informational purposes and does not constitute formal legal counsel.

To learn more, read Apify's guide: [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/)

# Actor input Schema

## `mode` (type: `string`):

search returns one row per dataset. resources returns one row per file attached to those datasets.

## `query` (type: `string`):

Text passed to the CKAN package_search q parameter on data.gov.uk. Example: health.

## `filterQuery` (type: `string`):

Optional CKAN fq filter, such as organization:cabinet-office. Leave empty to search the whole catalogue. A rejected filter is saved as an error row.

## `includeResources` (type: `boolean`):

In search mode, attach the dataset file list on each row. Ignored in resources mode, which already emits one row per file.

## `maxItems` (type: `integer`):

Maximum dataset or resource rows to save. Values above 1000 are clamped to 1000.

## `timeoutSecs` (type: `integer`):

Stop fetching when this many seconds have passed. Already saved rows stay in the dataset.

## `requestTimeoutSecs` (type: `integer`):

Seconds to wait for each data.gov.uk HTTP response before retrying.

## `proxyConfiguration` (type: `object`):

Optional proxy for CKAN requests. Default is no proxy. The public API does not require one, and a proxy is not used to bypass stated limits.

## Actor input object example

```json
{
  "mode": "search",
  "query": "health",
  "filterQuery": "organization:cabinet-office",
  "includeResources": true,
  "maxItems": 100,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `datasetUrl` (type: `string`):

Default dataset for this run, with one row per dataset, resource, or error.

## `overviewUrl` (type: `string`):

Table of creator, id, licence, and metadata timestamps.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "search",
    "query": "health",
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("b2b_lead_generation/data-gov-uk-dataset-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "search",
    "query": "health",
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("b2b_lead_generation/data-gov-uk-dataset-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "search",
  "query": "health",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call b2b_lead_generation/data-gov-uk-dataset-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,b2b_lead_generation/data-gov-uk-dataset-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nkLatFQM1wN0P3ugn/builds/XJuTfa3IpkgDSR0e9/openapi.json