# DataONE Member/Coordinating Node Extractor (`datamule/dataone-member-node-extractor`) Actor

Point at ANY DataONE member node or the coordinating node (KNB, Arctic, ESS-DIVE, PANGAEA, TDAR) and extract structured rows: search 1.6M+ research datasets (title/author/coverage), list the ~85-node federation registry, or dump a node's object holdings.

- **URL**: https://apify.com/datamule/dataone-member-node-extractor.md
- **Developed by:** [Datamule](https://apify.com/datamule) (community)
- **Categories:** Developer tools, Business
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 data records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## DataONE Member/Coordinating Node Extractor

Point at **any DataONE member node (MN) or the coordinating node (CN)** and get clean, structured rows. DataONE is a federation of ~85 research-data repositories — KNB, the Arctic Data Center, ESS-DIVE, PANGAEA, LTER, TDAR, USGS ScienceBase, Dryad and more — that all speak the **same** MN/CN REST API v2. So there is no per-repository scraper: one actor reads the whole federation.

### What it does

Three modes, one output schema:

| Mode | Endpoint | One row per… |
|------|----------|--------------|
| **search** (default) | `GET {cnUrl}/v2/query/solr/?q=…&wt=json` | research dataset — id, title, author, origin, keywords, abstract, formatId, spatial bounding box (N/S/E/W), temporal coverage (begin/end), source node, dates |
| **nodes** | `GET {cnUrl}/v2/node` | registered node — identifier (`urn:node:KNB`), name, description, baseURL, type (mn/cn), state, service inventory, contact, geolocation, operator properties |
| **objects** | `GET {nodeUrl}/v2/object` | object (pid) held by one node — identifier, formatId, size, checksum, `dateSysMetadataModified` |

The CN Solr index holds **1.6M+** indexed science-metadata records; individual nodes hold hundreds of thousands to over a million objects each (KNB ~314k, Arctic ~1.18M).

### Input

```json
{
  "mode": "search",
  "cnUrl": "https://cn.dataone.org/cn",
  "query": "formatType:METADATA",
  "maxRecords": 1000
}
```

- `mode` — `search` | `nodes` | `objects` (default `search`)
- `cnUrl` — coordinating node base URL (default is the production CN)
- `query` — a Solr query for `search` mode (e.g. `*:*`, `title:salmon`, `keywords:temperature AND northBoundCoord:[* TO *]`)
- `nodeUrl` — a member-node base URL for `objects` mode. **Path prefixes vary** — take the full `baseURL` verbatim (KNB `/knb/d1/mn`, Arctic `/metacat/d1/mn`, ESS-DIVE `/catalog/d1/mn`). Run `nodes` mode first to discover them.
- `fromDate` / `toDate` — optional `dateSysMetadataModified` bounds for `objects` mode (ISO 8601)
- `maxRecords` — cap on records (paginated automatically; default 1000)

#### Discover-then-extract

Run **nodes** to inventory the federation and read each operator's `baseURL`, then feed a `baseURL` into **objects** mode to walk that node's holdings. **search** goes straight to the CN's cross-federation discovery index.

### Output

Every row carries `_mode`, `_rowIndex`, and the lossless `_raw` source record so you can re-derive any field. Optional fields are nullable — nodes without geolocation, datasets without a bounding box, or objects without sysmeta dates come back as `null` rather than being dropped.

### Notes

- The wire format is per-endpoint: `/node` and `/object` return XML; `/query/solr` returns JSON (with `wt=json`). This actor parses each correctly.
- Some member nodes require a client certificate for `/object` (CN-only). Those return an authorization error and the run fails loudly — it never emits an empty result to hide a failure. The seeded public nodes are anonymous-readable.
- Pricing is **pay-per-event**: one charge per record returned.

### Data source

DataONE MN/CN REST API v2 — <https://purl.dataone.org/architecture/apis/MN_APIs.html>. Please respect each node's terms of use.

# Actor input Schema

## `mode` (type: `string`):

What to extract. 'search' (default) queries the coordinating node's Solr index for research datasets (title/author/abstract/keywords/spatial+temporal coverage). 'nodes' lists every registered node in the federation (the ~85-operator registry). 'objects' lists the raw object holdings (pids) of ONE member node — set 'Member node URL' for this mode.

## `cnUrl` (type: `string`):

The DataONE coordinating node base URL, used by 'search' and 'nodes' modes. The production CN is the default (85 nodes, 1.6M+ indexed datasets). Point at another environment's CN if needed.

## `query` (type: `string`):

A DataONE Solr query for 'search' mode. Examples: '*:*' (everything), 'formatType:METADATA', 'title:salmon', 'keywords:temperature AND northBoundCoord:\[\* TO *]', 'datasource:urn:node:KNB'. Defaults to '*:\*'.

## `nodeUrl` (type: `string`):

A member-node base URL for 'objects' mode — take the full baseURL verbatim (path prefixes vary): KNB https://knb.ecoinformatics.org/knb/d1/mn · Arctic https://arcticdata.io/metacat/d1/mn · ESS-DIVE https://data.ess-dive.lbl.gov/catalog/d1/mn. Run 'nodes' mode first to discover baseURLs. Leave empty to use the coordinating node.

## `fromDate` (type: `string`):

Optional lower bound on dateSysMetadataModified for 'objects' mode (ISO 8601, e.g. 2024-01-01T00:00:00Z).

## `toDate` (type: `string`):

Optional upper bound on dateSysMetadataModified for 'objects' mode (ISO 8601).

## `maxRecords` (type: `integer`):

Maximum number of records to extract (paginated automatically). Nodes mode is capped by the registry size (~85).

## `userAgent` (type: `string`):

Override the default User-Agent header sent to the node.

## `timeoutSecs` (type: `integer`):

Per-request timeout in seconds.

## Actor input object example

```json
{
  "mode": "search",
  "cnUrl": "https://cn.dataone.org/cn",
  "query": "formatType:METADATA",
  "nodeUrl": "https://knb.ecoinformatics.org/knb/d1/mn",
  "maxRecords": 1000,
  "timeoutSecs": 90
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "cnUrl": "https://cn.dataone.org/cn",
    "query": "formatType:METADATA"
};

// Run the Actor and wait for it to finish
const run = await client.actor("datamule/dataone-member-node-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "cnUrl": "https://cn.dataone.org/cn",
    "query": "formatType:METADATA",
}

# Run the Actor and wait for it to finish
run = client.actor("datamule/dataone-member-node-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "cnUrl": "https://cn.dataone.org/cn",
  "query": "formatType:METADATA"
}' |
apify call datamule/dataone-member-node-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=datamule/dataone-member-node-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/hgXwWbnwBinelfPD5/builds/eprTwCegoJMxtbLhh/openapi.json