OGC API Records Extractor

A generic "point at any OGC API - Records service" scraper. OGC API - Records is the modern, JSON/GeoJSON, OGC-blessed catalog / metadata standard — the REST successor to the XML/SOAP CSW (Catalogue Service for the Web). One open specification spoken by the pygeoapi, pycsw and ldproxy server families and by national metadata catalogs. Give this actor a service root URL, optionally pick a catalog + an area

• a time window + a free-text query, and it pages the whole record /items collection into clean, flat dataset rows plus a lossless raw GeoJSON Feature.

One actor, every OGC API - Records catalog — no per-instance scraper needed. (It's the discovery/metadata sibling of the OGC API Features Extractor and the STAC Catalog Extractor: Features and STAC give you the data, Records gives you the catalog that describes it.)

What it does

• Fetch records — set the service root + one or more catalog collection ids and any of a free-text search (q), bounding box, date/time window, or an arbitrary query parameters passthrough (type / externalId / CQL filter / provider-specific keys), and it pulls every matching record, auto-following the rel="next" pagination link (relative or absolute) up to your Max records.
• Discover catalogs — leave the collection id(s) empty (or turn on List collections only) and it reads {serviceRoot}/collections and returns one row per record catalog (a collection with itemType: "record"): id, title, description, keywords, spatial + temporal extent, CRS, and a ready-to-use items URL.
• Normalized + raw — each record is distilled to stable columns (recordId, title, description, type, created, updated, keywords, themeConcepts, externalIds, language, geometryType, a computed bbox, and longitude/latitude for point records) while the full properties, geometry and the untouched Feature (raw) are all preserved, so nothing is lost even though every catalog exposes slightly different metadata.

Example inputs

Metadata records from the pygeoapi reference server (no auth):

{ "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": ["dutch-metadata"], "maxRecords": 100 }

Free-text search across a catalog:

{ "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": ["dutch-metadata"], "q": "kaart", "maxRecords": 50 }

Records from a pycsw catalog (no auth):

{ "serviceRoot": "https://demo.pycsw.org/gisdata", "collectionId": ["metadata:main"], "maxRecords": 100 }

Records inside a bounding box:

{ "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": ["dutch-metadata"], "bbox": [3, 50, 8, 54], "maxRecords": 50 }

Discover which record catalogs a service exposes:

{ "serviceRoot": "https://demo.pygeoapi.io/master", "listCollectionsOnly": true }

A key-gated deployment — supply your own token, never hardcoded:

{ "serviceRoot": "https://your-catalog.example.org", "collectionId": ["records"], "bearerToken": "YOUR_TOKEN", "maxRecords": 100 }

Input reference

• serviceRoot (required) — the OGC API root (not /collections or /items).
• collectionId — record-catalog collection id(s) to fetch; empty ⇒ discovery mode.
• listCollectionsOnly — force discovery mode (list record catalogs, fetch nothing).
• q — free-text search over titles/descriptions/keywords.
• bbox — [west, south, east, north] in WGS84 degrees (a 6-number 3D box is also accepted).
• datetime — RFC 3339 instant or range; open ranges with ...
• queryParams — arbitrary extra /items query params (type, externalId, CQL filter, …).
• limit — server-side page size (the actor paginates regardless).
• maxRecords — total record cap across all pages/catalogs (default 1000).
• bearerToken / extraHeaders — optional auth for key-gated deployments.

Authentication

None required for the public catalogs (pygeoapi, pycsw demos). For key-gated deployments supply a Bearer token and/or Extra request headers — both optional, never required, and the token is never logged.

Output

One dataset item per record, with normalized columns plus:

• _serviceRoot / _mode — the source service and whether the row is a record or a collection.
• title / description / type / created / updated / language — the well-known OGC Records metadata fields (any absent ones are null).
• keywords / themeConcepts / externalIds — flattened keyword, theme-concept and external-identifier lists.
• geometry — the record footprint (GeoJSON), properties — the full metadata bag, bbox — computed from the geometry when the record omits one, longitude/latitude — set for point records.
• raw — the complete, untouched GeoJSON Feature.

A query that matches nothing returns 0 records, not an error.

Pricing

Pay per record — you're charged one record event per row returned (one per catalog in discovery mode). A query that matches nothing costs nothing.

Notes

• Speaks OGC API - Records Part 1 (Core) record access; handles the GeoJSON record /items collection and the standard rel="next" offset pagination (relative and absolute hrefs). A catalog collection self-identifies with itemType: "record" and its records carry conformsTo: […/ogcapi-records-1/1.0/conf/record-core].
• Not affiliated with or endorsed by the OGC or any target service. Respect each provider's terms and any per-dataset licensing.