ESGF Search Harvester — CMIP6 / CORDEX Climate Data

Point this Actor at any node in the Earth System Grid Federation (ESGF) and harvest climate-model dataset and file records through the federation's uniform, Solr-backed Search API. It's a generic runner: the same query grammar works across every ESGF index node — LLNL, DKRZ, IPSL, CEDA, NCI and the rest — so you can target one node or the default federation trio without changing anything but the nodes list.

Built for climate scientists, earth-system modellers and research-data engineers who need programmatic, structured access to CMIP6 / CMIP5 / CORDEX / obs4MIPs / input4MIPs holdings without hand-rolling the Solr query layer.

What it does

Three modes, auto-selected from the mode input:

Facet filters

Filter by the standard ESGF facets, exposed as first-class inputs (all optional, all multi-value):

project · experiment_id · variable · frequency · source_id (model) · institution_id · activity_id · variant_label · grid_label · realm · table_id · data_node — plus a free-text query and an extraFacets object for anything else the node indexes.

Example input

{
  "nodes": ["https://esgf-node.llnl.gov/esg-search"],
  "mode": "search",
  "type": "Dataset",
  "project": ["CMIP6"],
  "experiment_id": ["historical"],
  "variable": ["tas"],
  "frequency": ["mon"],
  "latestOnly": true,
  "maxRecords": 200
}

Download-manifest example (wget mode)

ESGF download scripts list files only, so in wget mode the Actor always asks the node for file records (type=File), whatever the Record type field says — the untouched default Dataset just works. All your facet filters are kept exactly as entered. Rows are file entries (filename, url, checksum), one charged record each.

{
  "nodes": ["https://esgf-data.dkrz.de/esg-search"],
  "mode": "wget",
  "project": ["CMIP6"],
  "experiment_id": ["historical"],
  "variable": ["tas"],
  "frequency": ["mon"],
  "maxRecords": 6
}

Example output (search / Dataset)

{
  "type": "Dataset",
  "id": "CMIP6.CMIP.EC-Earth-Consortium.EC-Earth3.historical.r23i1p1f1.Amon.tas.gr.v20210527|eagle.alcf.anl.gov",
  "title": "CMIP6.CMIP.EC-Earth-Consortium.EC-Earth3.historical.r23i1p1f1.Amon.tas.gr",
  "project": ["CMIP6"],
  "source_id": ["EC-Earth3"],
  "experiment_id": ["historical"],
  "variable": ["tas"],
  "frequency": ["mon"],
  "institution_id": ["EC-Earth-Consortium"],
  "variant_label": ["r23i1p1f1"],
  "data_node": "eagle.alcf.anl.gov",
  "size": 782045502
}

Multi-valued Solr fields (variable, project, url, checksum, …) are preserved as arrays — never flattened to the first element. The complete raw Solr document is kept under raw so nothing is lost.

Notes on the federation

• format=application/solr+json is pinned for you — without it ESGF returns XML.
• distrib defaults to false (query only the target node). Set it true to fan the query out across the whole federation — more coverage, but slower and can partially fail.
• latestOnly (default true) restricts to the latest dataset version; replica controls whether replica copies are included.
• Pagination is automatic (numFound + offset/limit); set maxRecords: 0 to harvest a result set to completion (can be very large — CMIP6 has tens of millions of records).

Pricing

Pay-per-result: you're charged per record returned. Discovery runs (facets mode, or a small maxRecords) are cheap; full harvests scale with what you pull.

Your max run charge is respected. Set a maximum charge per run and the Actor delivers only as many records as that budget pays for, then stops before making any further request to ESGF nodes. Every record in the dataset is exactly one charged record — no unpaid extras, no double charges if the run restarts. A max charge below the price of one record returns 0 records and costs nothing. The wget mode only reads the download script's file list; it never runs the script or downloads climate data.

About the ESGF Search API

The ESGF Search API is a thin REST facade over a federated Apache Solr index. Every node speaks the same grammar, which is exactly what makes one generic Actor work across the whole federation. Reference: ESGF Search RESTful API.