ESGF Search Harvester — CMIP6 / CORDEX Climate Data

Point this Actor at any node in the Earth System Grid Federation (ESGF) and harvest climate-model dataset and file records through the federation's uniform, Solr-backed Search API. It's a generic runner: the same query grammar works across every ESGF index node — LLNL, DKRZ, IPSL, CEDA, NCI and the rest — so you can target one node or the default federation trio without changing anything but the nodes list.

Built for climate scientists, earth-system modellers and research-data engineers who need programmatic, structured access to CMIP6 / CMIP5 / CORDEX / obs4MIPs / input4MIPs holdings without hand-rolling the Solr query layer.

What it does

Three modes, auto-selected from the mode input:

Facet filters

Filter by the standard ESGF facets, exposed as first-class inputs (all optional, all multi-value):

project · experiment_id · variable · frequency · source_id (model) · institution_id · activity_id · variant_label · grid_label · realm · table_id · data_node — plus a free-text query and an extraFacets object for anything else the node indexes.

Example input

{
  "nodes": ["https://esgf-node.llnl.gov/esg-search"],
  "mode": "search",
  "type": "Dataset",
  "project": ["CMIP6"],
  "experiment_id": ["historical"],
  "variable": ["tas"],
  "frequency": ["mon"],
  "latestOnly": true,
  "maxRecords": 200
}

Example output (search / Dataset)

{
  "type": "Dataset",
  "id": "CMIP6.CMIP.EC-Earth-Consortium.EC-Earth3.historical.r23i1p1f1.Amon.tas.gr.v20210527|eagle.alcf.anl.gov",
  "title": "CMIP6.CMIP.EC-Earth-Consortium.EC-Earth3.historical.r23i1p1f1.Amon.tas.gr",
  "project": ["CMIP6"],
  "source_id": ["EC-Earth3"],
  "experiment_id": ["historical"],
  "variable": ["tas"],
  "frequency": ["mon"],
  "institution_id": ["EC-Earth-Consortium"],
  "variant_label": ["r23i1p1f1"],
  "data_node": "eagle.alcf.anl.gov",
  "size": 782045502
}

Multi-valued Solr fields (variable, project, url, checksum, …) are preserved as arrays — never flattened to the first element. The complete raw Solr document is kept under raw so nothing is lost.

Notes on the federation

• format=application/solr+json is pinned for you — without it ESGF returns XML.
• distrib defaults to false (query only the target node). Set it true to fan the query out across the whole federation — more coverage, but slower and can partially fail.
• latestOnly (default true) restricts to the latest dataset version; replica controls whether replica copies are included.
• Pagination is automatic (numFound + offset/limit); set maxRecords: 0 to harvest a result set to completion (can be very large — CMIP6 has tens of millions of records).

Pricing

Pay-per-result: you're charged per record returned. Discovery runs (facets mode, or a small maxRecords) are cheap; full harvests scale with what you pull.

About the ESGF Search API

The ESGF Search API is a thin REST facade over a federated Apache Solr index. Every node speaks the same grammar, which is exactly what makes one generic Actor work across the whole federation. Reference: ESGF Search RESTful API.