eu-dataset-files

The actual downloadable files behind EU open datasets — one flat row per file, with the direct URL, the format, the licence and the attribution duty already on the row.

The portal answers with datasets; a pipeline needs files. A single dataset in our test run had 15 distributions across 13 formats, and finding the CSV among them means walking a nested record per hit. This unit does that walk for you.

Input

Every field is optional.

{
  "query": "water",
  "country": ["be"],
  "catalogue": ["data-gov-be"],
  "categories": ["ENVI"],
  "formats": ["CSV"],
  "language": "en",
  "max_datasets": 20,
  "max_files": 100,
  "require_download_url": true
}

Field	Meaning
formats	narrows the files too, not just the datasets — ask for CSV and you get CSV rows, not every format of every matching dataset
max_datasets	how many datasets to walk (1–200)
max_files	hard cap on rows returned (1–1000)
require_download_url	true (default) returns only files with a direct link. Set false to also get files that offer a landing page instead

Output

{
  "unit": "eu-dataset-files",
  "version": "1.0.0",
  "total_matching_datasets": 11017,
  "datasets_scanned": 2,
  "returned": 1,
  "files": [
    {
      "dataset_id": "7eacade4-ff0d-4df6-a988-df2f64c42859",
      "dataset_title": "...",
      "dataset_portal_url": "https://data.europa.eu/data/datasets/...?locale=en",
      "catalogue_id": "data-gov-gr",
      "country_code": "GR",
      "publisher": null,
      "file_id": "393fa372-bc7e-450f-87fb-aeb5aa4f8930",
      "file_title": "...",
      "format": "CSV",
      "media_type": null,
      "download_url": "https://data.gov.gr/dataset/.../download/00013.csv",
      "access_url": "https://data.gov.gr/dataset/.../download/00013.csv",
      "license": { "id": "CC_BY_4_0", "label": "Creative Commons Attribution 4.0 International",
                   "uri": "http://spdx.org/licenses/CC-BY-4.0" },
      "attribution_required": "yes",
      "modified": "2026-04-28T13:34:39.293151"
    }
  ]
}

total_matching_datasets is the size of the haystack; datasets_scanned is how much of it was walked under your caps. When they differ, the file list is a sample, not the whole set.

The licence is per file, not per dataset — the portal attaches it to the distribution, and datasets do mix licences across their files. That is why attribution_required is on every row: whichever file you download, the duty that comes with it is on the same line.

This unit returns links, it does not fetch the files. Downloading is your side of the line, and the rate limit of the host serving the file is yours to respect.

esquema.json is the contract. If it changes, this is a new version, not a fix.

Licensing, personal data, rate limiting

Portal metadata is CC0 1.0 (https://data.europa.eu/en/legal-notice); each file carries its own licence. No personal data is emitted — the upstream contact_point and creator fields never enter the output, and e-mail addresses are redacted from free text. One request per second, Retry-After honoured, official API over plain HTTP, no browser.

Running it

python main.py '{"query": "water", "formats": ["CSV"], "max_files": 10}'
python conferir.py output.json
python fumaca.py