OAI-PMH Repository Harvester

Point at any OAI-PMH endpoint and get clean, flat metadata rows — plus the lossless raw XML of every record. One actor, the entire open-repository world.

OAI-PMH (the Open Archives Initiative Protocol for Metadata Harvesting) is the metadata-interoperability standard of the scholarly, library, cultural-heritage and institutional-repository world. Every conforming provider — no matter what software it runs (DSpace, EPrints, Invenio, OJS, or a custom stack) — answers the identical six-verb HTTP grammar and returns the same <OAI-PMH> XML envelope. Anonymous read is the whole point of the protocol, so this single actor works against the ~4,000–8,000 repositories indexed by OpenDOAR / ROAR / BASE. No per-repository scraper, no host enumeration — just the base URL.

What you can harvest

Pick a verb and point sources at one or more OAI base URLs:

Verb	What you get (one row per…)
ListRecords (default)	full metadata record — identifier, datestamp, sets, title, creators, subjects, description, publisher, date, type, rights, DC identifiers + raw XML
ListIdentifiers	lightweight header — identifier, datestamp, sets, deleted flag
Identify	repository census — name, base URL, protocol version, admin emails, granularity, earliest datestamp, deleted-record policy
ListSets	a harvestable set — setSpec, setName
ListMetadataFormats	a supported format — prefix, schema, namespace
GetRecord	a single record by identifier

Why one actor covers thousands of sources

The wire format varies in exactly the ways a naive parser trips on — and this runner handles all of them:

• Namespaces & prefixes differ. arXiv serves <oai_dc:dc><dc:title>; another provider uses an un-prefixed <dc><title> under a default namespace. Every element is matched by its local name, so prefixes and namespace URIs never matter.
• resumptionToken pagination. Large result sets are returned page by page; the actor follows the server's cursor automatically up to your global maxRecords cap.
• Incremental harvesting. from / until datestamp windows (either YYYY-MM-DD or full ISO granularity) let you pull only what changed — OAI's native delta harvest.
• Deleted records (status="deleted") are captured as deleted: true, never a crash.
• Flow control. Busy repositories that answer HTTP 503 + Retry-After are backed off politely.
• Non-DC schemas (MARCXML, MODS, …) still yield a structured fields map plus the lossless _raw — nothing is ever dropped.

Every optional field is nullable, so a record missing a publisher/date/language simply yields null for that column. Point it at several repositories at once — an unreachable or non-OAI host is skipped with a warning and the batch continues.

Example input

{
  "sources": ["https://export.arxiv.org/oai2"],
  "verb": "ListRecords",
  "metadataPrefix": "oai_dc",
  "set": "cs",
  "from": "2024-01-01",
  "maxRecords": 5000
}

Discover a repository first with verb: "Identify" (the census) or verb: "ListSets" (its harvestable sets), then harvest the set you want.

Example endpoints

https://export.arxiv.org/oai2 (arXiv) · https://dspace.mit.edu/oai/request (DSpace) · https://zenodo.org/oai2d (Invenio/Zenodo) · https://pub.uni-bielefeld.de/oai · https://api.archives-ouvertes.fr/oai/hal (HAL) · any EPrints / OJS / DSpace / Invenio install.

Output

One flat row per item with a shared, fully-nullable column set (unused columns are null for the chosen verb) plus a lossless _raw XML string and _source / _verb provenance. Results are a clean table in the Console and available as JSON, CSV or Excel via the dataset API.

Pricing

Pay per result — you are charged one record event per row emitted. No subscription, no platform-usage surprises. An empty selective harvest (noRecordsMatch) costs nothing.

Notes

• Public OAI-PMH repositories require no authentication — harvesting is the protocol's purpose. An optional bearer token / extra headers can be supplied for gated deployments; they are never logged.
• Please harvest responsibly: use from / until for incremental updates rather than re-pulling a whole repository, and respect each provider's stated metadata policy.