OAI-PMH Repository Harvester

Point at any OAI-PMH endpoint and get clean, flat metadata rows — plus the lossless raw XML of every record. One actor, the entire open-repository world.

OAI-PMH (the Open Archives Initiative Protocol for Metadata Harvesting) is the metadata-interoperability standard of the scholarly, library, cultural-heritage and institutional-repository world. Every conforming provider — no matter what software it runs (DSpace, EPrints, Invenio, OJS, or a custom stack) — answers the identical six-verb HTTP grammar and returns the same <OAI-PMH> XML envelope. Anonymous read is the whole point of the protocol, so this single actor works against the ~4,000–8,000 repositories indexed by OpenDOAR / ROAR / BASE. No per-repository scraper, no host enumeration — just the base URL.

What you can harvest

Pick a verb and point sources at one or more OAI base URLs:

Verb	What you get (one row per…)
ListRecords	full metadata record — identifier, datestamp, sets, title, creators, subjects, description, publisher, date, type, rights, DC identifiers + raw XML
ListIdentifiers	lightweight header — identifier, datestamp, sets, deleted flag
Identify (starter example)	repository census — name, base URL, protocol version, admin emails, granularity, earliest datestamp, deleted-record policy
ListSets	a harvestable set — setSpec, setName
ListMetadataFormats	a supported format — prefix, schema, namespace
GetRecord	a single record by identifier

Why one actor covers thousands of sources

The wire format varies in exactly the ways a naive parser trips on — and this runner handles all of them:

• Namespaces & prefixes differ. arXiv serves <oai_dc:dc><dc:title>; another provider uses an un-prefixed <dc><title> under a default namespace. Every element is matched by its local name, so prefixes and namespace URIs never matter.
• resumptionToken pagination. Large result sets are returned page by page; the actor follows the server's cursor automatically up to your global maxRecords cap.
• Incremental harvesting. from / until datestamp windows (either YYYY-MM-DD or full ISO granularity) let you pull only what changed — OAI's native delta harvest.
• Deleted records (status="deleted") are captured as deleted: true, never a crash.
• Flow control. Busy repositories that answer HTTP 503 + Retry-After are backed off politely.
• Non-DC schemas (MARCXML, MODS, …) still yield a structured fields map plus the lossless _raw — nothing is ever dropped.

Every optional field is nullable, so a record missing a publisher/date/language simply yields null for that column. Point it at several repositories at once — an unreachable or non-OAI host is skipped with a warning and the batch continues.

Example input

{
  "sources": ["https://export.arxiv.org/oai2"],
  "verb": "ListRecords",
  "metadataPrefix": "oai_dc",
  "set": "cs",
  "from": "2024-01-01",
  "maxRecords": 5000
}

Discover a repository first with verb: "Identify" (the census) or verb: "ListSets" (its harvestable sets), then harvest the set you want.

Example endpoints

https://export.arxiv.org/oai2 (arXiv) · https://dspace.mit.edu/oai/request (DSpace) · https://zenodo.org/oai2d (Invenio/Zenodo) · https://pub.uni-bielefeld.de/oai · https://api.archives-ouvertes.fr/oai/hal (HAL) · any EPrints / OJS / DSpace / Invenio install.

Output

One flat row per item with a shared, fully-nullable column set (unused columns are null for the chosen verb) plus a lossless _raw XML string and _source / _verb provenance. Results are a clean table in the Console and available as JSON, CSV or Excel via the dataset API.

Pricing

Pay per result — you are charged one record event per row emitted. No subscription, no platform-usage surprises. An empty selective harvest (noRecordsMatch) costs nothing.

Your max run charge is respected. Each row is billed as it is written, and the harvester checks your remaining budget before every repository request. If the budget cannot pay for one more record, it stops cleanly (status: Max run charge reached) instead of delivering unbilled rows.

Notes

• Public OAI-PMH repositories require no authentication — harvesting is the protocol's purpose. An optional bearer token / extra headers can be supplied for gated deployments; they are never logged.
• Please harvest responsibly: use from / until for incremental updates rather than re-pulling a whole repository, and respect each provider's stated metadata policy.