DataONE Member/Coordinating Node Extractor

Point at any DataONE member node (MN) or the coordinating node (CN) and get clean, structured rows. DataONE is a federation of ~85 research-data repositories — KNB, the Arctic Data Center, ESS-DIVE, PANGAEA, LTER, TDAR, USGS ScienceBase, Dryad and more — that all speak the same MN/CN REST API v2. So there is no per-repository scraper: one actor reads the whole federation.

What it does

Three modes, one output schema:

Mode	Endpoint	One row per…
search (default)	GET {cnUrl}/v2/query/solr/?q=…&wt=json	research dataset — id, title, author, origin, keywords, abstract, formatId, spatial bounding box (N/S/E/W), temporal coverage (begin/end), source node, dates
nodes	GET {cnUrl}/v2/node	registered node — identifier (urn:node:KNB), name, description, baseURL, type (mn/cn), state, service inventory, contact, geolocation, operator properties
objects	GET {nodeUrl}/v2/object	object (pid) held by one node — identifier, formatId, size, checksum, dateSysMetadataModified

The CN Solr index holds 1.6M+ indexed science-metadata records; individual nodes hold hundreds of thousands to over a million objects each (KNB ~314k, Arctic ~1.18M).

Input

{
  "mode": "search",
  "cnUrl": "https://cn.dataone.org/cn",
  "query": "formatType:METADATA",
  "maxRecords": 1000
}

• mode — search | nodes | objects (default search)
• cnUrl — coordinating node base URL (default is the production CN)
• query — a Solr query for search mode (e.g. *:*, title:salmon, keywords:temperature AND northBoundCoord:[* TO *])
• nodeUrl — a member-node base URL for objects mode. Path prefixes vary — take the full baseURL verbatim (KNB /knb/d1/mn, Arctic /metacat/d1/mn, ESS-DIVE /catalog/d1/mn). Run nodes mode first to discover them.
• fromDate / toDate — optional dateSysMetadataModified bounds for objects mode (ISO 8601)
• maxRecords — cap on records (paginated automatically; default 1000)

Discover-then-extract

Run nodes to inventory the federation and read each operator's baseURL, then feed a baseURL into objects mode to walk that node's holdings. search goes straight to the CN's cross-federation discovery index.

Output

Every row carries _mode, _rowIndex, and the lossless _raw source record so you can re-derive any field. Optional fields are nullable — nodes without geolocation, datasets without a bounding box, or objects without sysmeta dates come back as null rather than being dropped.

Notes

• The wire format is per-endpoint: /node and /object return XML; /query/solr returns JSON (with wt=json). This actor parses each correctly.
• Some member nodes require a client certificate for /object (CN-only). Those return an authorization error and the run fails loudly — it never emits an empty result to hide a failure. The seeded public nodes are anonymous-readable.
• Pricing is pay-per-event: one charge per record returned.

Data source

DataONE MN/CN REST API v2 — https://purl.dataone.org/architecture/apis/MN_APIs.html. Please respect each node's terms of use.