Ireland Open Data Scraper - data.gov.ie Portal
Pricing
from $6.23 / 1,000 datasets
Scrape all 22,666 datasets from Ireland's open data portal data.gov.ie: full DCAT metadata, distributions, 174 publishers, EU high-value flags, openness scores and DataStore column schemas.
Free-text search across the portal. Several terms are combined with OR. Fielded Solr syntax also works, for example title:housing or notes:census. Leave empty to sweep the whole portal.
[]Individual data.gov.ie dataset pages, for example https://data.gov.ie/dataset/planning-permission. When any URL or slug is given the portal sweep is skipped and only those datasets are returned.
[]CKAN dataset names (planning-permission) or UUIDs. Same effect as Dataset URLs, without typing the host.
[]Free users: limited to 10 items (preview). Paid users: up to 1,000,000. Counts every row the run writes, datasets and directory rows together.
Portal themes. Measured live: Government 11,725 datasets, Environment 4,228, Health 2,551, Justice 765.
[]CKAN organization names, for example central-statistics-office (12,849 datasets), meteireann (2,051), tusla (1,492), marine-institute (1,064). Tick Publisher directory below to pull the full list of 174.
[]The upstream feed a dataset was harvested from. cso_jsonrpc holds 13,780 datasets, isde 1,914, tusla-ckan 1,492, dublinked-ckan 924.
[]Keep only datasets that publish at least one file in these formats. The index is case sensitive and carries 132 spellings, so each choice expands to every spelling the portal actually uses.
[]The 12 licence ids in the portal. CC-BY-4.0 covers 21,925 datasets, CC-BY-SA-4.0 429, cc-by-nc-nd 56, cc-zero 28.
[]How often the publisher says the dataset changes. Monthly 2,600 datasets, Quarterly 882, Annual 513, Daily 40.
[]The six categories of Implementing Regulation (EU) 2023/138. Geospatial covers 4,322 datasets, Companies 113, Mobility 78, Meteorological 46.
[]CKAN groups. The portal currently runs exactly one, haleandhearty, with 431 datasets.
[]Metadata language. 22,665 datasets are tagged en and 1 is tagged ga.
Keep only datasets with at least this many downloadable files. 1,354 datasets carry 5 or more.
Keep datasets whose metadata changed on or after this date. 5,912 datasets changed since 2026-01-01.
Keep datasets whose metadata changed on or before this date.
Keep datasets first registered on the portal on or after this date. 4,655 arrived since 2025-01-01.
Keep datasets first registered on the portal on or before this date.
Keep datasets the publisher issued on or after this date. 1,151 were issued since 2024-01-01.
Keep datasets the publisher issued on or before this date.
Keep only the 383 datasets flagged as high value under Implementing Regulation (EU) 2023/138.
Keep only the 4,009 datasets that carry a spatial geometry. The portal's bounding-box search is broken upstream, so this flag is the geographic filter it offers.
Keep only the 21,628 datasets the portal marks as openly licensed.
Keep only the 4,378 datasets that carry a full rights or reuse statement beyond the licence id.
Keep only the 1,960 datasets that declare a standard they conform to, such as INSPIRE or DCAT.
Raw Solr fq clause ANDed with everything above, for example num_tags:[10 TO *]. Use it for fields this schema does not expose.
Result order. Only these values are honoured; the portal silently falls back to relevance for anything else.
Write one extra row for every downloadable file, with its format, size, checksum and direct URL. Datasets average 3.4 files each.
Write one row for each of the 174 publishers, with acronym, description, dataset count and follower count.
Write one row per portal theme with its live dataset count. 14 themes.
Write one row per EU high-value category with its live dataset count. 6 categories.
Write one row per licence with its live dataset count. 12 licences.
Write one row per declared update frequency with its live dataset count. 15 values.
Write one row per harvest collection with its live dataset count. 27 collections.
Write one row per file-format spelling with its live dataset count. 132 spellings.
Write one row per keyword tag with its live dataset count. 22,742 tags, so raise Max Items before ticking this.
Write one row per upstream feed the portal harvests, with its URL, harvester type, schedule and next run. 31 sources.
Write one row per showcase, the applications and stories the portal features on top of its data. 14 showcases.
Write one row per public data request, what a citizen or company asked a body to publish and whether it was accepted. 112 requests, 9 still open.
Write one row per recent dataset change on the portal, with the timestamp and the dataset it touched. Use it to monitor what moved since your last run.
Fetch the portal's own 0 to 5 star openness rating and the reason behind it for every dataset. Costs one extra API call per dataset, so it is off by default.
For files loaded into the portal's DataStore (19.4% of them), read the real column names, per-column SQL types, the true row count and a sample of rows. Nothing in the metadata carries this.
Send a HEAD request to each file URL and report the status, final URL, content type and byte size. The portal harvests 31 external feeds and never revalidates the links.
Attach the publisher's acronym, description, dataset count and follower count to every dataset row. Profiles are cached, so one call per publisher, not per dataset.
How many files per dataset the link check and the DataStore probe are allowed to touch. Raising it raises both the run time and the cost.