# OAI-PMH Repository Harvester (`datamule/oai-pmh-repository-harvester`) Actor Point at ANY OAI-PMH endpoint (arXiv, DSpace, EPrints, Invenio, OJS and thousands more) and get flat metadata rows — identifier, datestamp, title, creators, subjects, publisher, date, rights — plus lossless raw XML. All 6 verbs, incremental from/until, resumptionToken paging. Pay per record. - **URL**: https://apify.com/datamule/oai-pmh-repository-harvester.md - **Developed by:** [Datamule](https://apify.com/datamule) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.35 / 1,000 records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## OAI-PMH Repository Harvester Point at **any OAI-PMH endpoint** and get clean, flat metadata rows — plus the lossless raw XML of every record. One actor, the entire open-repository world. **OAI-PMH** (the Open Archives Initiative Protocol for Metadata Harvesting) is *the* metadata-interoperability standard of the scholarly, library, cultural-heritage and institutional-repository world. Every conforming provider — no matter what software it runs (**DSpace, EPrints, Invenio, OJS**, or a custom stack) — answers the identical six-verb HTTP grammar and returns the same `` XML envelope. Anonymous read is the whole point of the protocol, so this single actor works against the **~4,000–8,000 repositories** indexed by OpenDOAR / ROAR / BASE. No per-repository scraper, no host enumeration — just the base URL. ### What you can harvest Pick a **verb** and point `sources` at one or more OAI base URLs: | Verb | What you get (one row per…) | |------|------------------------------| | **ListRecords** *(default)* | full metadata record — identifier, datestamp, sets, title, creators, subjects, description, publisher, date, type, rights, DC identifiers + raw XML | | **ListIdentifiers** | lightweight header — identifier, datestamp, sets, deleted flag | | **Identify** | repository census — name, base URL, protocol version, admin emails, granularity, earliest datestamp, deleted-record policy | | **ListSets** | a harvestable set — setSpec, setName | | **ListMetadataFormats** | a supported format — prefix, schema, namespace | | **GetRecord** | a single record by identifier | ### Why one actor covers thousands of sources The wire format varies in exactly the ways a naive parser trips on — and this runner handles all of them: - **Namespaces & prefixes differ.** arXiv serves ``; another provider uses an un-prefixed `` under a default namespace. **Every element is matched by its local name**, so prefixes and namespace URIs never matter. - **resumptionToken pagination.** Large result sets are returned page by page; the actor follows the server's cursor automatically up to your global `maxRecords` cap. - **Incremental harvesting.** `from` / `until` datestamp windows (either `YYYY-MM-DD` or full ISO granularity) let you pull only what changed — OAI's native delta harvest. - **Deleted records** (`status="deleted"`) are captured as `deleted: true`, never a crash. - **Flow control.** Busy repositories that answer `HTTP 503 + Retry-After` are backed off politely. - **Non-DC schemas** (MARCXML, MODS, …) still yield a structured `fields` map plus the lossless `_raw` — nothing is ever dropped. Every optional field is nullable, so a record missing a publisher/date/language simply yields `null` for that column. Point it at several repositories at once — an unreachable or non-OAI host is skipped with a warning and the batch continues. ### Example input ```json { "sources": ["https://export.arxiv.org/oai2"], "verb": "ListRecords", "metadataPrefix": "oai_dc", "set": "cs", "from": "2024-01-01", "maxRecords": 5000 } ```` Discover a repository first with `verb: "Identify"` (the census) or `verb: "ListSets"` (its harvestable sets), then harvest the set you want. ### Example endpoints `https://export.arxiv.org/oai2` (arXiv) · `https://dspace.mit.edu/oai/request` (DSpace) · `https://zenodo.org/oai2d` (Invenio/Zenodo) · `https://pub.uni-bielefeld.de/oai` · `https://api.archives-ouvertes.fr/oai/hal` (HAL) · any EPrints / OJS / DSpace / Invenio install. ### Output One flat row per item with a shared, fully-nullable column set (unused columns are `null` for the chosen verb) plus a lossless `_raw` XML string and `_source` / `_verb` provenance. Results are a clean table in the Console and available as JSON, CSV or Excel via the dataset API. ### Pricing **Pay per result** — you are charged one `record` event per row emitted. No subscription, no platform-usage surprises. An empty selective harvest (`noRecordsMatch`) costs nothing. ### Notes - Public OAI-PMH repositories require **no authentication** — harvesting is the protocol's purpose. An optional bearer token / extra headers can be supplied for gated deployments; they are never logged. - Please harvest responsibly: use `from` / `until` for incremental updates rather than re-pulling a whole repository, and respect each provider's stated metadata policy. # Actor input Schema ## `sources` (type: `array`): One or more OAI-PMH base URLs — the Open Archives Initiative Protocol for Metadata Harvesting endpoint of ANY conforming repository (DSpace, EPrints, Invenio, OJS, custom). A bare base URL is fine — the actor appends the ?verb query itself. Examples: https://export.arxiv.org/oai2 (arXiv), https://dspace.mit.edu/oai/request (MIT DSpace), https://pub.uni-bielefeld.de/oai (Bielefeld), https://zenodo.org/oai2d (Zenodo/Invenio), https://api.archives-ouvertes.fr/oai/hal (HAL). One actor works against the ~4,000–8,000 repositories indexed by OpenDOAR/ROAR/BASE. ## `verb` (type: `string`): The OAI-PMH verb to run. ListRecords (default) harvests full metadata records; ListIdentifiers is the lightweight header-only harvest (identifier + datestamp + sets); Identify returns the repository census row; ListSets lists the harvestable sets; ListMetadataFormats lists the supported metadata formats; GetRecord fetches a single record by identifier. ## `metadataPrefix` (type: `string`): The metadata format to request (ListRecords / ListIdentifiers / GetRecord). oai\_dc (Dublin Core) is the spec-mandated default present on every conforming provider and is flattened to clean columns (title / creators / subjects / description / publisher / date / type / rights …). Other formats (marcxml, mods, oai\_datacite, …) still yield a structured fields map + the lossless raw XML. Not every provider supports every format beyond oai\_dc. ## `set` (type: `string`): Restrict the harvest to one OAI set (a setSpec, e.g. cs on arXiv for Computer Science). Run the actor once with verb=ListSets to discover a repository's sets. Leave empty to harvest the whole repository. ## `from` (type: `string`): Incremental harvest lower bound — only records with a datestamp on/after this date (OAI's native incremental harvesting). Format matches the repository's granularity: YYYY-MM-DD or a full ISO 8601 UTC timestamp (YYYY-MM-DDThh:mm:ssZ). Leave empty for no lower bound. ## `until` (type: `string`): Incremental harvest upper bound — only records with a datestamp on/before this date. Same format as From. Leave empty for no upper bound. ## `identifier` (type: `string`): The OAI record identifier for verb=GetRecord, e.g. oai:arXiv.org:cs/0208027. Required only for GetRecord; ignored otherwise. ## `maxRecords` (type: `integer`): A GLOBAL cap on the total number of records/rows to harvest across all sources and all resumptionToken pages (each emitted record is one billable event). Default 1000 (roughly one arXiv page). Increase to auto-paginate through a large result set. ## `bearer` (type: `string`): Optional bearer token for gated OAI deployments (sent as Authorization: Bearer \*\*\*). Not required for public repositories — anonymous read is the point of the protocol. Never logged. ## `extraHeaders` (type: `object`): Optional extra HTTP headers as a JSON object, e.g. {"x-api-key": "..."} for gated servers. Not required for public repositories. Header values are never logged. ## Actor input object example ```json { "sources": [ "https://export.arxiv.org/oai2" ], "verb": "ListRecords", "metadataPrefix": "oai_dc", "set": "cs", "maxRecords": 1000 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '<YOUR_API_TOKEN>' with your token const client = new ApifyClient({ token: '<YOUR_API_TOKEN>', }); // Prepare Actor input const input = { "sources": [ "https://export.arxiv.org/oai2" ], "metadataPrefix": "oai_dc", "set": "cs" }; // Run the Actor and wait for it to finish const run = await client.actor("datamule/oai-pmh-repository-harvester").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '<YOUR_API_TOKEN>' with your token. client = ApifyClient("<YOUR_API_TOKEN>") # Prepare the Actor input run_input = { "sources": ["https://export.arxiv.org/oai2"], "metadataPrefix": "oai_dc", "set": "cs", } # Run the Actor and wait for it to finish run = client.actor("datamule/oai-pmh-repository-harvester").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sources": [ "https://export.arxiv.org/oai2" ], "metadataPrefix": "oai_dc", "set": "cs" }' | apify call datamule/oai-pmh-repository-harvester --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=datamule/oai-pmh-repository-harvester", "--header", "Authorization: Bearer <YOUR_API_TOKEN>" ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "OAI-PMH Repository Harvester", "description": "Point at ANY OAI-PMH endpoint (arXiv, DSpace, EPrints, Invenio, OJS and thousands more) and get flat metadata rows — identifier, datestamp, title, creators, subjects, publisher, date, rights — plus lossless raw XML. All 6 verbs, incremental from/until, resumptionToken paging. Pay per record.", "version": "0.1", "x-build-id": "IW80fPXyDTeHvDI1d" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/datamule~oai-pmh-repository-harvester/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-datamule-oai-pmh-repository-harvester", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/datamule~oai-pmh-repository-harvester/runs": { "post": { "operationId": "runs-sync-datamule-oai-pmh-repository-harvester", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/datamule~oai-pmh-repository-harvester/run-sync": { "post": { "operationId": "run-sync-datamule-oai-pmh-repository-harvester", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "sources" ], "properties": { "sources": { "title": "OAI-PMH endpoint URL(s)", "type": "array", "description": "One or more OAI-PMH base URLs — the Open Archives Initiative Protocol for Metadata Harvesting endpoint of ANY conforming repository (DSpace, EPrints, Invenio, OJS, custom). A bare base URL is fine — the actor appends the ?verb query itself. Examples: https://export.arxiv.org/oai2 (arXiv), https://dspace.mit.edu/oai/request (MIT DSpace), https://pub.uni-bielefeld.de/oai (Bielefeld), https://zenodo.org/oai2d (Zenodo/Invenio), https://api.archives-ouvertes.fr/oai/hal (HAL). One actor works against the ~4,000–8,000 repositories indexed by OpenDOAR/ROAR/BASE.", "items": { "type": "string" } }, "verb": { "title": "Verb", "enum": [ "ListRecords", "ListIdentifiers", "Identify", "ListSets", "ListMetadataFormats", "GetRecord" ], "type": "string", "description": "The OAI-PMH verb to run. ListRecords (default) harvests full metadata records; ListIdentifiers is the lightweight header-only harvest (identifier + datestamp + sets); Identify returns the repository census row; ListSets lists the harvestable sets; ListMetadataFormats lists the supported metadata formats; GetRecord fetches a single record by identifier.", "default": "ListRecords" }, "metadataPrefix": { "title": "Metadata prefix", "type": "string", "description": "The metadata format to request (ListRecords / ListIdentifiers / GetRecord). oai_dc (Dublin Core) is the spec-mandated default present on every conforming provider and is flattened to clean columns (title / creators / subjects / description / publisher / date / type / rights …). Other formats (marcxml, mods, oai_datacite, …) still yield a structured fields map + the lossless raw XML. Not every provider supports every format beyond oai_dc.", "default": "oai_dc" }, "set": { "title": "Set (selective harvest)", "type": "string", "description": "Restrict the harvest to one OAI set (a setSpec, e.g. cs on arXiv for Computer Science). Run the actor once with verb=ListSets to discover a repository's sets. Leave empty to harvest the whole repository." }, "from": { "title": "From datestamp", "type": "string", "description": "Incremental harvest lower bound — only records with a datestamp on/after this date (OAI's native incremental harvesting). Format matches the repository's granularity: YYYY-MM-DD or a full ISO 8601 UTC timestamp (YYYY-MM-DDThh:mm:ssZ). Leave empty for no lower bound." }, "until": { "title": "Until datestamp", "type": "string", "description": "Incremental harvest upper bound — only records with a datestamp on/before this date. Same format as From. Leave empty for no upper bound." }, "identifier": { "title": "Record identifier (GetRecord)", "type": "string", "description": "The OAI record identifier for verb=GetRecord, e.g. oai:arXiv.org:cs/0208027. Required only for GetRecord; ignored otherwise." }, "maxRecords": { "title": "Max records (global cap)", "minimum": 1, "type": "integer", "description": "A GLOBAL cap on the total number of records/rows to harvest across all sources and all resumptionToken pages (each emitted record is one billable event). Default 1000 (roughly one arXiv page). Increase to auto-paginate through a large result set.", "default": 1000 }, "bearer": { "title": "Bearer token", "type": "string", "description": "Optional bearer token for gated OAI deployments (sent as Authorization: Bearer ***). Not required for public repositories — anonymous read is the point of the protocol. Never logged." }, "extraHeaders": { "title": "Extra request headers", "type": "object", "description": "Optional extra HTTP headers as a JSON object, e.g. {\"x-api-key\": \"...\"} for gated servers. Not required for public repositories. Header values are never logged." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```