data.gov.uk Dataset Scraper: UK Open Data

Search the official UK open data catalogue to extract comprehensive dataset records or attached resource files. Gather structured metadata—including dataset titles, publisher details, resource formats, and licence types—into clean datasets ready for instant export to JSON, CSV, or Excel.

Instead of manually browsing through catalogue pages, you can collect UK public sector data records in bulk for research, compliance, and analysis.

Use Cases

• Catalogue research: Search UK open data for specific topics (such as health or transport) and maintain structured metadata in a single table.
• Resource & file inventory: Switch to resources mode to list downloadable files (CSV, JSON, PDF) linked across datasets.
• Publisher filtering: Apply search filters to target specific government departments or organizations.
• Automated updates: Run scheduled queries to track changes and updates across the UK open data ecosystem.
• Licence compliance: Check dataset licensing details (license_title, license_url) before reusing attached files.

Extracted Data Points

Search Mode (Dataset Records)

• Dataset ID, title, and package type
• Description, notes, and publishing organization
• Primary and secondary themes
• Creation and modification timestamps
• Licence title, ID, and licence URL
• Attached resources list (when enabled)

Resources Mode (File Records)

• Resource ID, direct download URL, and file format (e.g., CSV, PDF, GeoJSON)
• Parent dataset ID, title, and publisher metadata

Input Parameters

Parameter	Type	Default	Description
mode	String	search	search exports one row per dataset. resources exports one row per attached file.
query	String	health	Search terms for the catalogue query.
filterQuery	String	empty	Optional filter query (e.g., organization:cabinet-office).
includeResources	Boolean	true	In search mode, includes the attached file list within each dataset record.
maxItems	Integer	100	Maximum number of rows to retrieve (up to 1,000).
timeoutSecs	Integer	300	Overall execution limit in seconds.

Example Input

{
    "mode": "search",
    "query": "health",
    "filterQuery": "",
    "includeResources": true,
    "maxItems": 100,
    "timeoutSecs": 300
}

Output Format

{
    "id": "d44b4f65-563a-4e3c-9a01-f51b4a4dbdd2",
    "isopen": true,
    "license_id": "uk-ogl",
    "license_title": "UK Open Government Licence (OGL)",
    "license_url": "https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/",
    "metadata_created": "2016-06-10T09:56:02.268810",
    "metadata_modified": "2026-07-08T13:33:16.922355",
    "name": "health",
    "notes": "Reference Costs provide indicators of the unit costs of providing a range of hospital and community services at trust and hospital level.",
    "organization": {
        "id": "8549022f-fa16-43e4-a56f-6a45ca875199",
        "name": "open-data-ni",
        "title": "OpenDataNI"
    },
    "theme-primary": "health",
    "datasetTitle": "Health Trust Reference Costs 2014-15",
    "datasetType": "dataset"
}

How It Works

1. Processes configuration inputs and sets search parameters or organization filters.
2. Pages through catalogue results efficiently while automatically skipping duplicate record IDs.
3. Manages temporary connection disruptions smoothly with retry delays.
4. Logs invalid query filters as error entries while allowing remaining queries to run smoothly.
5. Generates clean datasets available for download in JSON, CSV, or Excel formats.

Frequently Asked Questions

What is the difference between search mode and resources mode?

search mode outputs one record per dataset (including dataset metadata). resources mode outputs one record per individual file attached to those datasets, featuring direct file URLs and format labels.

What happens if a filter query is rejected or invalid?

An error entry explaining the issue is logged, while the process continues cleanly without crashing.

Can I export datasets directly to CSV or Excel?

Yes. Completed datasets can be downloaded instantly in JSON, CSV, or Excel formats from your run overview.

---

Legal & Compliance

Trademark Notice

data.gov.uk Dataset and related names, logos, and trademarks belong to their respective owners. Their mention in this documentation is strictly to identify data utility and compatibility, and does not imply affiliation, sponsorship, or endorsement.

Disclaimer

This tool is an independent utility and is not affiliated with, endorsed by, or officially connected to third-party public catalogue providers. Users are responsible for ensuring their usage complies with relevant terms of service, licence conditions, and guidelines.

Is web scraping legal?

This tool collects publicly available catalogue records without bypassing access controls. Collecting public data is standard practice, but proper compliance requires considering how the data is used:

• Personal data: Ensure dataset usage respects regional privacy regulations (such as GDPR or CCPA) when combining outputs with personal user records.
• Copyright & Terms: Respect the terms of use, usage limits, and licensing terms (such as the UK Open Government Licence) set by data sources.
• Legal advice: This summary is provided for general informational purposes and does not constitute formal legal counsel.

To learn more, read Apify's guide: Is web scraping legal?