# OGC API Records Extractor (`datamule/ogc-api-records-extractor`) Actor Point at ANY OGC API - Records catalog (the JSON/GeoJSON successor to CSW) — pygeoapi, pycsw, ldproxy. List record catalogs or page /items by bbox/datetime/free-text/type/CQL to flat rows (recordId, type, title, created, keywords, themes, bbox, lon/lat) + raw GeoJSON. Pay per record. - **URL**: https://apify.com/datamule/ogc-api-records-extractor.md - **Developed by:** [Datamule](https://apify.com/datamule) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.35 / 1,000 records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## OGC API Records Extractor A **generic "point at any OGC API - Records service" scraper**. [OGC API - Records](https://ogcapi.ogc.org/records/) is the modern, JSON/GeoJSON, OGC-blessed **catalog / metadata** standard — the REST successor to the XML/SOAP **CSW** (Catalogue Service for the Web). One open specification spoken by the **pygeoapi**, **pycsw** and **ldproxy** server families and by national metadata catalogs. Give this actor a service root URL, optionally pick a catalog + an area + a time window + a free-text query, and it pages the whole record `/items` collection into clean, flat dataset rows plus a lossless `raw` GeoJSON Feature. One actor, every OGC API - Records catalog — no per-instance scraper needed. (It's the discovery/metadata sibling of the [OGC API Features Extractor](https://apify.com/datamule/ogc-api-features-extractor) and the [STAC Catalog Extractor](https://apify.com/datamule/stac-catalog-extractor): Features and STAC give you the *data*, Records gives you the *catalog that describes it*.) ### What it does - **Fetch records** — set the service root + one or more catalog collection ids and any of a **free-text search** (`q`), **bounding box**, **date/time window**, or an arbitrary **query parameters** passthrough (`type` / `externalId` / CQL `filter` / provider-specific keys), and it pulls every matching record, auto-following the `rel="next"` pagination link (relative or absolute) up to your **Max records**. - **Discover catalogs** — leave the collection id(s) empty (or turn on **List collections only**) and it reads `{serviceRoot}/collections` and returns one row per **record catalog** (a collection with `itemType: "record"`): id, title, description, keywords, spatial + temporal extent, CRS, and a ready-to-use items URL. - **Normalized + raw** — each record is distilled to stable columns (`recordId`, `title`, `description`, `type`, `created`, `updated`, `keywords`, `themeConcepts`, `externalIds`, `language`, `geometryType`, a computed **`bbox`**, and **`longitude`/`latitude`** for point records) while the full `properties`, `geometry` and the untouched Feature (`raw`) are all preserved, so nothing is lost even though every catalog exposes slightly different metadata. ### Example inputs Metadata records from the pygeoapi reference server (no auth): ```json { "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": ["dutch-metadata"], "maxRecords": 100 } ```` Free-text search across a catalog: ```json { "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": ["dutch-metadata"], "q": "kaart", "maxRecords": 50 } ``` Records from a pycsw catalog (no auth): ```json { "serviceRoot": "https://demo.pycsw.org/gisdata", "collectionId": ["metadata:main"], "maxRecords": 100 } ``` Records inside a bounding box: ```json { "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": ["dutch-metadata"], "bbox": [3, 50, 8, 54], "maxRecords": 50 } ``` Discover which record catalogs a service exposes: ```json { "serviceRoot": "https://demo.pygeoapi.io/master", "listCollectionsOnly": true } ``` A key-gated deployment — supply your own token, never hardcoded: ```json { "serviceRoot": "https://your-catalog.example.org", "collectionId": ["records"], "bearerToken": "YOUR_TOKEN", "maxRecords": 100 } ``` ### Input reference - `serviceRoot` **(required)** — the OGC API root (not `/collections` or `/items`). - `collectionId` — record-catalog collection id(s) to fetch; empty ⇒ discovery mode. - `listCollectionsOnly` — force discovery mode (list record catalogs, fetch nothing). - `q` — free-text search over titles/descriptions/keywords. - `bbox` — `[west, south, east, north]` in WGS84 degrees (a 6-number 3D box is also accepted). - `datetime` — RFC 3339 instant or range; open ranges with `..`. - `queryParams` — arbitrary extra `/items` query params (`type`, `externalId`, CQL `filter`, …). - `limit` — server-side page size (the actor paginates regardless). - `maxRecords` — total record cap across all pages/catalogs (default 1000). - `bearerToken` / `extraHeaders` — optional auth for key-gated deployments. ### Authentication **None required** for the public catalogs (pygeoapi, pycsw demos). For key-gated deployments supply a **Bearer token** and/or **Extra request headers** — both optional, never required, and the token is never logged. ### Output One dataset item per record, with normalized columns plus: - `_serviceRoot` / `_mode` — the source service and whether the row is a `record` or a `collection`. - `title` / `description` / `type` / `created` / `updated` / `language` — the well-known OGC Records metadata fields (any absent ones are `null`). - `keywords` / `themeConcepts` / `externalIds` — flattened keyword, theme-concept and external-identifier lists. - `geometry` — the record footprint (GeoJSON), `properties` — the full metadata bag, `bbox` — computed from the geometry when the record omits one, `longitude`/`latitude` — set for point records. - `raw` — the complete, untouched GeoJSON Feature. A query that matches nothing returns **0 records, not an error**. ### Pricing **Pay per record** — you're charged one `record` event per row returned (one per catalog in discovery mode). A query that matches nothing costs nothing. ### Notes - Speaks OGC API - Records Part 1 (Core) record access; handles the GeoJSON record `/items` collection and the standard `rel="next"` offset pagination (relative and absolute hrefs). A catalog collection self-identifies with `itemType: "record"` and its records carry `conformsTo: […/ogcapi-records-1/1.0/conf/record-core]`. - Not affiliated with or endorsed by the OGC or any target service. Respect each provider's terms and any per-dataset licensing. # Actor input Schema ## `serviceRoot` (type: `string`): The root URL of ANY OGC API - Records service (the modern JSON/GeoJSON catalog/metadata standard — the REST successor to CSW). Examples: https://demo.pygeoapi.io/master (pygeoapi, its dutch-metadata records catalog), https://demo.pycsw.org/gisdata (pycsw). Do NOT include /collections or /items — just the landing/API root. ## `collectionId` (type: `array`): Record-catalog collection id(s) to pull records from, e.g. dutch-metadata (pygeoapi) or metadata:main (pycsw). A catalog collection self-identifies with itemType="record". Accepts one or more. Leave EMPTY to run in discovery mode (list every record catalog the service exposes). Turn on "List collections only" below to just browse what's available first. ## `listCollectionsOnly` (type: `boolean`): When on, the actor only reads {serviceRoot}/collections and returns one row per available record catalog (id, title, description, itemType, keywords, spatial/temporal extent, CRS, items URL) — it does NOT fetch any records. Use this to discover what a service exposes, then set Catalog collection id(s) + a query to extract records. ## `q` (type: `string`): OGC API - Records free-text query — matches record titles, descriptions and keywords. Example: kaart or climate. Leave empty for no text filter. (Honored by servers that implement the Records free-text search conformance class, e.g. pygeoapi, pycsw.) ## `bbox` (type: `array`): Spatial filter as \[west, south, east, north] in WGS84 decimal degrees (a 6-number \[w,s,minZ,e,n,maxZ] 3D box is also accepted). Example for the Netherlands: \[3, 50, 8, 54]. Records are matched by their footprint geometry. Leave empty for no spatial filter. ## `datetime` (type: `string`): Temporal filter in RFC 3339. A single instant (2024-06-15T00:00:00Z), a closed range (2024-06-01T00:00:00Z/2024-06-30T23:59:59Z), or an open range using ".." (2024-06-01T00:00:00Z/.. or ../2024-06-30T00:00:00Z). Leave empty for no time filter. ## `queryParams` (type: `object`): Arbitrary extra query-string parameters appended to each /items request, as a JSON object — the generic passthrough for OGC Records filters (type / externalId), CQL, and provider-specific keys. Examples: filter by record type {"type": "dataset"}; an external id {"externalId": "…"}; a CQL filter {"filter": "type='service'", "filter-lang": "cql2-text"}. The reserved keys bbox/datetime/q/limit/offset/f are managed by the actor and ignored here. Leave empty for none. ## `limit` (type: `integer`): Server-side page size requested per /items call (the actor auto-follows the rel="next" link regardless). Most servers cap this around 100–10000. Leave at the default unless you have a reason to change it. ## `maxRecords` (type: `integer`): Maximum total number of records to pull across all pages and catalogs (the actor auto-paginates until this cap or the result set is exhausted). The default total cap is 25 records. Each returned row is one `record` PPE event at the current FREE unit price of $0.0005, so the maximum default/example event charge falls to $0.0125, excluding Apify infrastructure cost. ## `bearerToken` (type: `string`): Optional OAuth2 bearer token for key-gated deployments. Sent as Authorization: Bearer \*\*\* — NOT required for public catalogs (pygeoapi, pycsw demos) — leave empty. Never logged. ## `extraHeaders` (type: `object`): Optional extra HTTP headers to send with every request, as a JSON object, e.g. {"key": "..."} for a service that takes an API key in a header. Leave empty for public catalogs. ## Actor input object example ```json { "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": [ "dutch-metadata" ], "listCollectionsOnly": false, "limit": 100, "maxRecords": 25 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": [ "dutch-metadata" ], "limit": 100, "maxRecords": 25 }; // Run the Actor and wait for it to finish const run = await client.actor("datamule/ogc-api-records-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": ["dutch-metadata"], "limit": 100, "maxRecords": 25, } # Run the Actor and wait for it to finish run = client.actor("datamule/ogc-api-records-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "serviceRoot": "https://demo.pygeoapi.io/master", "collectionId": [ "dutch-metadata" ], "limit": 100, "maxRecords": 25 }' | apify call datamule/ogc-api-records-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=datamule/ogc-api-records-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "OGC API Records Extractor", "description": "Point at ANY OGC API - Records catalog (the JSON/GeoJSON successor to CSW) — pygeoapi, pycsw, ldproxy. List record catalogs or page /items by bbox/datetime/free-text/type/CQL to flat rows (recordId, type, title, created, keywords, themes, bbox, lon/lat) + raw GeoJSON. Pay per record.", "version": "0.1", "x-build-id": "UaQjF7Fb5ziuSX0wc" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/datamule~ogc-api-records-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-datamule-ogc-api-records-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/datamule~ogc-api-records-extractor/runs": { "post": { "operationId": "runs-sync-datamule-ogc-api-records-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/datamule~ogc-api-records-extractor/run-sync": { "post": { "operationId": "run-sync-datamule-ogc-api-records-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "serviceRoot" ], "properties": { "serviceRoot": { "title": "OGC API - Records service root URL", "type": "string", "description": "The root URL of ANY OGC API - Records service (the modern JSON/GeoJSON catalog/metadata standard — the REST successor to CSW). Examples: https://demo.pygeoapi.io/master (pygeoapi, its dutch-metadata records catalog), https://demo.pycsw.org/gisdata (pycsw). Do NOT include /collections or /items — just the landing/API root." }, "collectionId": { "title": "Catalog collection id(s)", "type": "array", "description": "Record-catalog collection id(s) to pull records from, e.g. dutch-metadata (pygeoapi) or metadata:main (pycsw). A catalog collection self-identifies with itemType=\"record\". Accepts one or more. Leave EMPTY to run in discovery mode (list every record catalog the service exposes). Turn on \"List collections only\" below to just browse what's available first.", "items": { "type": "string" } }, "listCollectionsOnly": { "title": "List collections only (discovery)", "type": "boolean", "description": "When on, the actor only reads {serviceRoot}/collections and returns one row per available record catalog (id, title, description, itemType, keywords, spatial/temporal extent, CRS, items URL) — it does NOT fetch any records. Use this to discover what a service exposes, then set Catalog collection id(s) + a query to extract records.", "default": false }, "q": { "title": "Free-text search (q)", "type": "string", "description": "OGC API - Records free-text query — matches record titles, descriptions and keywords. Example: kaart or climate. Leave empty for no text filter. (Honored by servers that implement the Records free-text search conformance class, e.g. pygeoapi, pycsw.)" }, "bbox": { "title": "Bounding box (bbox)", "type": "array", "description": "Spatial filter as [west, south, east, north] in WGS84 decimal degrees (a 6-number [w,s,minZ,e,n,maxZ] 3D box is also accepted). Example for the Netherlands: [3, 50, 8, 54]. Records are matched by their footprint geometry. Leave empty for no spatial filter." }, "datetime": { "title": "Date/time window", "type": "string", "description": "Temporal filter in RFC 3339. A single instant (2024-06-15T00:00:00Z), a closed range (2024-06-01T00:00:00Z/2024-06-30T23:59:59Z), or an open range using \"..\" (2024-06-01T00:00:00Z/.. or ../2024-06-30T00:00:00Z). Leave empty for no time filter." }, "queryParams": { "title": "Extra query parameters", "type": "object", "description": "Arbitrary extra query-string parameters appended to each /items request, as a JSON object — the generic passthrough for OGC Records filters (type / externalId), CQL, and provider-specific keys. Examples: filter by record type {\"type\": \"dataset\"}; an external id {\"externalId\": \"…\"}; a CQL filter {\"filter\": \"type='service'\", \"filter-lang\": \"cql2-text\"}. The reserved keys bbox/datetime/q/limit/offset/f are managed by the actor and ignored here. Leave empty for none." }, "limit": { "title": "Page size", "minimum": 1, "type": "integer", "description": "Server-side page size requested per /items call (the actor auto-follows the rel=\"next\" link regardless). Most servers cap this around 100–10000. Leave at the default unless you have a reason to change it.", "default": 100 }, "maxRecords": { "title": "Max records (total)", "minimum": 1, "type": "integer", "description": "Maximum total number of records to pull across all pages and catalogs (the actor auto-paginates until this cap or the result set is exhausted). The default total cap is 25 records. Each returned row is one `record` PPE event at the current FREE unit price of $0.0005, so the maximum default/example event charge falls to $0.0125, excluding Apify infrastructure cost.", "default": 25 }, "bearerToken": { "title": "Bearer token (optional)", "type": "string", "description": "Optional OAuth2 bearer token for key-gated deployments. Sent as Authorization: Bearer *** — NOT required for public catalogs (pygeoapi, pycsw demos) — leave empty. Never logged." }, "extraHeaders": { "title": "Extra request headers", "type": "object", "description": "Optional extra HTTP headers to send with every request, as a JSON object, e.g. {\"key\": \"...\"} for a service that takes an API key in a header. Leave empty for public catalogs." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```