SRU Catalog Harvester

Point at ANY SRU/SRW library catalog server and get flat, structured bibliographic rows — one actor for thousands of library catalogs.

SRU (Search/Retrieve via URL) is the Library of Congress' standard protocol for querying bibliographic catalogs over plain HTTP, together with its sibling SRW. It is spoken by a huge population of national, university and union library catalogs — the German National Library (DNB), the Library of Congress, the Bibliothèque nationale de France (BnF), the Koninklijke Bibliotheek (KB-NL), the National Library of Australia's Trove, the K10plus / GBV union catalog, and hundreds more. Every one of them answers the identical operation=searchRetrieve grammar and returns the same <searchRetrieveResponse> envelope.

Every other "library" or "catalog" scraper wraps one endpoint. This actor is the generic runner: give it any SRU server's base URL and a CQL query, and it turns the matching records into dataset rows — cleanly flattened columns plus a lossless copy of each original record.

What it does

• searchRetrieve (default): runs your CQL query against base_url, reads the server's total (numberOfRecords), and paginates via startRecord until the result set is exhausted or your global maxRecords cap is reached. Each record becomes one flat row.
• explain (discovery): fetches the server's self-description and emits one row per searchable index (name, set, title, searchable/sortable flags) plus a server-info row (title, host, database, available record schemas). A cheap way to learn what a server supports before querying it.

Robust to the ways SRU servers differ in practice:

• Namespaces & prefixes vary — DNB serves the envelope on the default namespace, the Library of Congress prefixes it zs:, K10plus uses the OASIS SRU-2.0 namespace. Every element is matched by its local name, so this is handled transparently.
• Record schemas vary — Dublin Core (oai_dc / dc) is flattened to clean columns; MARCXML is decoded from the standard MARC tags (245 title, 1xx/700 authors, 26x imprint, 020 ISBN, 6xx subjects); any other schema (MODS, ESE, …) still yields a structured fields map. A lossless raw string always preserves the complete original record XML.
• SRU 1.1 / 1.2 / 2.0 are all supported.

A query that matches nothing returns 0 records and exits cleanly; a server-side diagnostic (e.g. an unsupported index) is surfaced as a warning, never a crash.

Input

Field	Type	Description
base_url (required)	string	The SRU/SRW endpoint, e.g. https://services.dnb.de/sru/dnb (DNB), http://lx2.loc.gov:210/lcdb (Library of Congress), https://sru.k10plus.de/gvk (K10plus).
query	string	A CQL search expression, e.g. dc.title=climate, dc.creator=goethe, or a bare keyword. Required for searchRetrieve. Run explain first to discover a server's indexes.
operation	string	searchRetrieve (default) or explain (describe the server).
recordSchema	string	Metadata schema to request: oai_dc / dc, marcxml / MARC21-xml, mods, … Leave empty for the server default.
maximumRecords	integer	Records per SRU page (most servers cap at 100). Default 50.
maxRecords	integer	Global cap on total records harvested across all pages. Leave empty to fetch a single page (cost-safe).
startRecord	integer	1-based position of the first record (SRU is 1-indexed). Default 1.
sruVersion	string	1.1 (default), 1.2 or 2.0.
bearer	string (secret)	Optional bearer token for gated deployments. Never logged.
extraHeaders	object	Optional extra HTTP headers, e.g. {"x-api-key": "…"}. Never logged.

Output

One dataset item per record. A Dublin Core result from the DNB, for example:

{
  "_recordSchema": "oai_dc",
  "_recordPosition": 1,
  "_base_url": "https://services.dnb.de/sru/dnb",
  "_query": "dc.title=climate",
  "_numberOfRecords": 48992,
  "title": "SDG 13: Climate Action",
  "creator": "Leal Filho, Walter [Herausgeber]",
  "creators": ["Leal Filho, Walter [Herausgeber]", "Scherenberg, Viviane [Herausgeber]"],
  "publisher": "Springer Spektrum",
  "date": "2027",
  "subject": ["550 Geowissenschaften"],
  "identifier": "978-3-662-72215-2",
  "identifiers": ["978-3-662-72215-2", "3-662-72215-1", "1373318015"],
  "language": "eng",
  "raw": "<dc xmlns:dc=\"http://purl.org/dc/elements/1.1/\">…</dc>"
}

Missing fields are null; raw always preserves the complete original record XML so nothing is lost.

Finding servers

Registries of SRU/SRW endpoints include the Library of Congress' SRU implementors list and each national library's developer pages. Any endpoint that answers operation=explain works with this actor.

Pricing

Pay-per-event: $0.0005 per record (one emitted dataset row). Leave maxRecords empty for a single cost-safe page, or set it to harvest a larger result set.

Notes & limits

• Scope is the SRU/SRW search protocol returning XML records (Dublin Core, MARCXML, and other XML schemas via the generic fields + raw view).
• Binary formats (e.g. MARC-in-ISO2709 delivered as a base64 blob) are preserved in raw but not decoded.
• A malformed base_url, a non-XML response, or an HTTP 4xx returns a clear error; an empty result set simply yields 0 records.