SRU Catalog Harvester

Point at ANY SRU/SRW library catalog server and get flat, structured bibliographic rows — one actor for thousands of library catalogs.

SRU (Search/Retrieve via URL) is the Library of Congress' standard protocol for querying bibliographic catalogs over plain HTTP, together with its sibling SRW. It is spoken by a huge population of national, university and union library catalogs — the German National Library (DNB), the Library of Congress, the Bibliothèque nationale de France (BnF), the Koninklijke Bibliotheek (KB-NL), the National Library of Australia's Trove, the K10plus / GBV union catalog, and hundreds more. Every one of them answers the identical operation=searchRetrieve grammar and returns the same <searchRetrieveResponse> envelope.

Every other "library" or "catalog" scraper wraps one endpoint. This actor is the generic runner: give it any SRU server's base URL and a CQL query, and it turns the matching records into dataset rows — cleanly flattened columns plus a lossless copy of each original record.

What it does

• searchRetrieve (default): runs your CQL query against base_url, reads the server's total (numberOfRecords), and paginates via startRecord until the result set is exhausted or your global maxRecords cap is reached. Each record becomes one flat row.
• explain (discovery): fetches the server's self-description and emits one row per searchable index (name, set, title, searchable/sortable flags) plus a server-info row (title, host, database, available record schemas). A cheap way to learn what a server supports before querying it.

Robust to the ways SRU servers differ in practice:

• Namespaces & prefixes vary — DNB serves the envelope on the default namespace, the Library of Congress prefixes it zs:, K10plus uses the OASIS SRU-2.0 namespace. Every element is matched by its local name, so this is handled transparently.
• Record schemas vary — Dublin Core (oai_dc / dc) is flattened to clean columns; MARCXML is decoded from the standard MARC tags (245 title, 1xx/700 authors, 26x imprint, 020 ISBN, 6xx subjects); any other schema (MODS, ESE, …) still yields a structured fields map. A lossless raw string always preserves the complete original record XML.
• SRU 1.1 / 1.2 / 2.0 are all supported.

A query that genuinely matches nothing returns 0 records and the run succeeds. If the server rejects the search instead, the run fails and its status says why — no empty "success" to mistake for a real answer, and nothing is charged.

If a run fails with "The SRU server rejected the search", the status shows the server's own message plus one fix:

• Unsupported index — the query uses an index this server doesn't have. Run once with operation = explain to list its indexes (e.g. dc.title= on DNB, pica.tit= on K10plus), then run again.
• Unknown schema for retrieval — change Record schema to one the server lists, or leave it empty.
• Unsupported parameter — remove that parameter from the pasted endpoint link (put access tokens in Bearer token instead).
• First record position out of range — lower Start record.

If a later page is rejected, the records already saved stay in the dataset (and were charged), but the run is marked failed / incomplete, so you know you don't have the full result.

Input

Field	Type	Description
base_url (required)	string	The SRU/SRW endpoint, e.g. https://services.dnb.de/sru/dnb (DNB), http://lx2.loc.gov:210/lcdb (Library of Congress), https://sru.k10plus.de/gvk (K10plus). You can also paste a full search link copied from your browser (...?operation=searchRetrieve&query=...&recordSchema=...): its query, record schema, version, page size and start position fill any form field you left empty or at its default; a value you changed in the form wins, and the log shows what was used.
query	string	A CQL search expression, e.g. dc.title=climate, dc.creator=goethe, or a bare keyword. Required for searchRetrieve. Run explain first to discover a server's indexes.
operation	string	searchRetrieve (default) or explain (describe the server).
recordSchema	string	Metadata schema to request: oai_dc / dc, marcxml / MARC21-xml, mods, … Leave empty for the server default.
maximumRecords	integer	Records per SRU page (most servers cap at 100). Default 50.
maxRecords	integer	Global cap on total records harvested across all pages. Leave empty to fetch a single page (cost-safe).
startRecord	integer	1-based position of the first record (SRU is 1-indexed). Default 1.
sruVersion	string	1.1 (default), 1.2 or 2.0.
bearer	string (secret)	Optional bearer token for gated deployments. Never logged.
extraHeaders	object	Optional extra HTTP headers, e.g. {"x-api-key": "…"}. Never logged.

Output

One dataset item per record. A Dublin Core result from the DNB, for example:

{
  "_recordSchema": "oai_dc",
  "_recordPosition": 1,
  "_base_url": "https://services.dnb.de/sru/dnb",
  "_query": "dc.title=climate",
  "_numberOfRecords": 48992,
  "title": "SDG 13: Climate Action",
  "creator": "Leal Filho, Walter [Herausgeber]",
  "creators": ["Leal Filho, Walter [Herausgeber]", "Scherenberg, Viviane [Herausgeber]"],
  "publisher": "Springer Spektrum",
  "date": "2027",
  "subject": ["550 Geowissenschaften"],
  "identifier": "978-3-662-72215-2",
  "identifiers": ["978-3-662-72215-2", "3-662-72215-1", "1373318015"],
  "language": "eng",
  "raw": "<dc xmlns:dc=\"http://purl.org/dc/elements/1.1/\">…</dc>"
}

Missing fields are null; raw always preserves the complete original record XML so nothing is lost.

Finding servers

Registries of SRU/SRW endpoints include the Library of Congress' SRU implementors list and each national library's developer pages. Any endpoint that answers operation=explain works with this actor.

Pricing

Pay-per-event: $0.0005 per record (one emitted dataset row). Leave maxRecords empty for a single cost-safe page, or set it to harvest a larger result set.

Your max run charge is respected. The Actor writes only the records your run's maximum charge covers, billing each one as it is written. Each SRU page request is sized to the records you can still afford, and it stops before requesting a page you have not paid for. If the budget does not cover even one record, no request is made and nothing is billed.

Notes & limits

• Scope is the SRU/SRW search protocol returning XML records (Dublin Core, MARCXML, and other XML schemas via the generic fields + raw view).
• Binary formats (e.g. MARC-in-ISO2709 delivered as a base64 blob) are preserved in raw but not decoded.
• A malformed base_url, a non-XML response, or an HTTP 4xx returns a clear error; an empty result set (numberOfRecords=0) is a successful run with 0 records; a server rejection fails the run with the server's message.
• Access tokens in a pasted link stay out of your results. If your endpoint link carries a credential (for example ?accessToken=..., ?apikey=... or user:password@), the Actor still sends it to the server, but the _base_url column, the run log and error messages show REDACTED in its place, so you can share the CSV/JSON export. Apify keeps the run's input as you entered it (visible to you in the run's Input tab); for a token that should never appear there, prefer the secret bearer field when your server supports it.