Research Repository Harvester — Dataverse / DSpace / InvenioRDM

Point at any Dataverse, DSpace 7/8, or InvenioRDM (Zenodo) install and harvest every dataset's metadata through one unified API. Three of the world's most widely-deployed research-data / institutional-repository software platforms each expose a single uniform REST API that every institution's deployment answers identically — so one actor spans thousands of independent repositories, and you get the same normalized output no matter which platform or install you point it at.

• Dataverse — the Harvard/IQSS software behind 150+ registered installs (Harvard, QDR, DaRUS Stuttgart, Borealis Canada, DataverseNL, …).
• DSpace 7/8 — the most-deployed institutional-repository software (theses, ETDs, gray literature) with 2,600+ installs worldwide.
• InvenioRDM / Zenodo — the CERN/Zenodo platform (Zenodo, Caltech Data, CERN).

What you get

Every dataset / item is normalized into one flat, consistent row — the same columns whether it came from Dataverse, DSpace or Invenio — plus a lossless raw copy of the entire original object so nothing is ever dropped:

Field	Description
repoId	The record's native id (Dataverse global id, DSpace UUID, Invenio record id)
platform	dataverse · dspace · invenio
baseUrl	The install the record came from
persistentId	DOI (Dataverse / Invenio) or Handle (DSpace)
title	Dataset / item title
authors	List of author / creator names
description	Abstract / description
subjects	Subjects + keywords
publicationDate	Publication / issued date
type	Resource type (dataset, thesis, publication, …)
license	Licence / rights
fileCount	Number of files (where the API exposes it)
landingPage	Canonical landing / DOI URL
raw	The complete original platform object (lossless)

Fields that a given platform or record doesn't provide come back as null — the schema is fully nullable, so a partial record never breaks a run.

Input

Field	Required	Description
platform	✅	dataverse, dspace, or invenio
baseUrl	✅	Install root — e.g. https://dataverse.harvard.edu, https://dspace.mit.edu, https://zenodo.org
query		Free-text search (empty = browse everything)
maxRecords		Max records to return (default: one page)
pageSize		Records per API page, 1–100 (default 25)
bearerToken		Optional token for a gated/private install (never required for public search, never logged)
extraHeaders		Optional extra HTTP headers as a JSON object

Examples

Zenodo (Invenio):

{ "platform": "invenio", "baseUrl": "https://zenodo.org", "query": "climate", "maxRecords": 100 }

Harvard Dataverse:

{ "platform": "dataverse", "baseUrl": "https://dataverse.harvard.edu", "query": "climate", "maxRecords": 100 }

MIT DSpace (point at the site root — the actor appends /server/api):

{ "platform": "dspace", "baseUrl": "https://dspace.mit.edu", "query": "climate", "maxRecords": 100 }

How it works

For each platform the actor calls its uniform search endpoint, pages through the result set, and maps each record into the unified schema:

• Dataverse → GET {base}/api/search?q=…&type=dataset&per_page=N&start=M (paginated by record offset), with a {base}/api/info/version handshake.
• DSpace → GET {base}/server/api/discover/search/objects?query=…&page=N&size=M (HAL/JSON envelope; Dublin-Core dc.* metadata flattened), with a {base}/server/api handshake that reports dspaceName / dspaceVersion.
• Invenio → GET {base}/api/records?q=…&size=M&page=N; handles both the legacy Zenodo record shape and the modern InvenioRDM shape transparently.

It returns catalog metadata only (what datasets exist and where to get them), never the underlying data files. A valid install with no matches returns 0 records and exits cleanly; a non-repository host (an HTML anti-bot page, a wrong URL, or a status:ERROR envelope) fails fast with a clear error rather than a silent empty run. No authentication is required for public repository search.

Pricing

Pay-per-event: you are charged per repository record returned. No monthly fee — you only pay for the records you harvest.