# SensorThings API Extractor (`datamule/sensorthings-api-extractor`) Actor Point at ANY OGC SensorThings API v1.1 instance (FROST-Server, GOST, SensorUp and thousands of air-quality, smart-city and research deployments) and get flat sensor rows — observations, things with coordinates, datastreams with units — plus raw JSON. @iot.nextLink paging, OData filters. Pay per row. - **URL**: https://apify.com/datamule/sensorthings-api-extractor.md - **Developed by:** [Datamule](https://apify.com/datamule) (community) - **Categories:** Developer tools, Business - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.35 / 1,000 records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## SensorThings API Extractor Point at **any OGC SensorThings API v1.1 instance** and get clean, flat sensor rows — plus the lossless raw JSON of every entity. One actor, the entire open sensor-data world. The **OGC SensorThings API** (OGC 15-078r6 / 18-088) is the open standard for Internet-of-Things and environmental-sensor data on the web. Every conforming deployment — no matter what server it runs (**FROST-Server, GOST, SensorUp**, or a custom stack) — answers the identical RESTful JSON grammar over the same entity graph, and anonymous read is the norm (publishing sensor data is the protocol's purpose). So this single actor works against **every SensorThings instance on Earth** — air-quality networks, smart-city platforms, government monitoring, university research. No per-instance scraper, no host enumeration — just the base URL. ### What you can extract Pick a **mode** and point `sources` at one or more SensorThings base URLs: | Mode | What you get (one row per…) | |------|------------------------------| | **observations** *(default)* | sensor reading — `@iot.id`, phenomenon time, result time, `result` (any type), result quality, parameters, valid time + Datastream / FeatureOfInterest links | | **things** | monitored asset — id, name, description, properties, and its Location's `coordinates` (Locations auto-expanded) | | **datastreams** | measurement series — name, observation type, unit of measurement, observed area, phenomenon/result time + Sensor / ObservedProperty / Thing links | | **discovery** | instance fingerprint — available entity sets, declared conformance classes, server settings | The SensorThings entity graph is `Things → Locations → Datastreams → (Sensor, ObservedProperty) → Observations`. Run **discovery** first to fingerprint an instance, **things** or **datastreams** to map what it monitors, then **observations** (optionally scoped to a single `datastreamId`) to pull the readings. ### Why one actor covers thousands of sources The wire format is uniform, but installs populate different **optional** fields — and this runner handles all of them: - **Every field is read by key presence.** A Thing without `properties`, a Datastream without an `observedArea`, an Observation without `resultQuality` or `parameters` → that column is simply `null`, never a crash. - **`@iot.nextLink` cursor pagination.** Large collections are returned page by page; the actor follows the server's cursor automatically up to your global `maxRecords` cap. - **Full OData query surface.** Pass `$filter`, `$expand`, `$select`, `$orderby` straight through — e.g. `$filter=result gt 0`, `$orderby=phenomenonTime desc`, `$expand=Datastream,FeatureOfInterest`. - **`result` is any type.** A reading may be a number, a string, a boolean, an array, or a complex object (OM_ComplexObservation) — it is passed through verbatim. - **Coordinates promoted.** A Thing's Point location and a Datastream's observed area are promoted to a clean `[lon, lat]` `coordinates` column, with the full GeoJSON kept in `geometry` / `_raw`. ### Example input ```json { "sources": ["https://iot.hamburg.de/v1.1"], "mode": "observations", "$orderby": "phenomenonTime desc", "maxRecords": 1000 } ```` Map an instance's assets instead: ```json { "sources": ["https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1"], "mode": "things" } ``` ### Output Every row is flat and fully nullable, with the same column set across all modes (unused columns are `null`), plus a lossless `_raw` copy of the source entity and `_source` / `_mode` / `_entity` provenance. Results stream to the dataset and export as JSON, CSV, Excel, or via the API. ### Resilience & honesty - A **reachable instance with no matching rows** → 0 records, clean success. Nothing is ever fabricated. - An **unreachable / DNS-failed host, an auth gate (401/403), or a non-SensorThings body** is skipped with a warning and the batch continues; a run where **every** source is skipped fails fast so you know your input was wrong rather than getting a silent empty result. - **No authentication is required** for public instances. An optional bearer token / extra headers can be supplied for gated deployments — they are never logged. ### Pricing Pay-per-event: you are charged **per record** emitted to the dataset. Use `maxRecords` (and, for observations, `datastreamId` or a `$filter`) to bound cost precisely. # Actor input Schema ## `sources` (type: `array`): One or more OGC SensorThings API v1.1 base URLs — the endpoint of ANY conforming instance (FROST-Server, GOST, SensorUp, …). A bare base URL is fine; the actor appends the entity-set path (/Things, /Datastreams, /Observations) itself, and a URL ending in an entity set is normalized back to the base. Examples: https://iot.hamburg.de/v1.1 (City of Hamburg), https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1 (Fraunhofer IOSB air quality), https://sensorthings.imk-ifu.kit.edu/v1.1 (KIT TERENO). One actor works against every OGC-compliant SensorThings deployment on Earth. ## `mode` (type: `string`): Which entity to harvest. observations (default) returns sensor readings (result + timestamps); things returns the monitored assets with their locations/coordinates; datastreams returns the measurement series (unit of measurement, observed area, phenomenon time) linking a Thing to a Sensor and ObservedProperty; discovery returns one fingerprint row per instance (available entity sets + declared conformance classes + server settings). ## `datastreamId` (type: `string`): Optional. In observations mode, scope the harvest to a single datastream by fetching /Datastreams({id})/Observations instead of the whole /Observations collection. Get datastream ids by running the actor once in datastreams mode. Leave empty to harvest all observations on the instance. ## `$filter` (type: `string`): Optional OData filter passed through verbatim to the SensorThings query, e.g. result gt 0, phenomenonTime gt 2024-01-01T00:00:00Z, name eq 'PM10', or startswith(name,'Air'). Leave empty for no filter. See the OGC SensorThings API v1.1 spec for the full $filter grammar. ## `$expand` (type: `string`): Optional OData expand passed through verbatim, e.g. Datastream,FeatureOfInterest on observations, or Sensor,ObservedProperty,Thing on datastreams, to inline related entities. In things mode this defaults to Locations (so each Thing carries its coordinates) unless you override it here. ## `$select` (type: `string`): Optional OData select passed through verbatim to narrow the fields the server returns, e.g. @iot.id,result,phenomenonTime. Leave empty to return the full entity (the actor always keeps the raw entity in the \_raw column). ## `$orderby` (type: `string`): Optional OData orderby passed through verbatim, e.g. phenomenonTime desc to get the newest observations first. Leave empty for the server default order. ## `maxRecords` (type: `integer`): A GLOBAL cap on the total number of rows to harvest across all sources and all @iot.nextLink pages (each emitted row is one billable event). Default 1000. Increase to auto-paginate through a large collection; observations collections can be very large, so this is your cost control. ## `bearer` (type: `string`): Optional bearer token for gated SensorThings deployments (sent as Authorization: Bearer …). Not required for public instances — anonymous read is the norm for the protocol. Never logged. ## `extraHeaders` (type: `object`): Optional extra HTTP headers as a JSON object, e.g. {"x-api-key": "..."} for gated servers. Not required for public instances. Header values are never logged. ## Actor input object example ```json { "sources": [ "https://iot.hamburg.de/v1.1" ], "mode": "observations", "maxRecords": 1000 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sources": [ "https://iot.hamburg.de/v1.1" ] }; // Run the Actor and wait for it to finish const run = await client.actor("datamule/sensorthings-api-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sources": ["https://iot.hamburg.de/v1.1"] } # Run the Actor and wait for it to finish run = client.actor("datamule/sensorthings-api-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sources": [ "https://iot.hamburg.de/v1.1" ] }' | apify call datamule/sensorthings-api-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=datamule/sensorthings-api-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "SensorThings API Extractor", "description": "Point at ANY OGC SensorThings API v1.1 instance (FROST-Server, GOST, SensorUp and thousands of air-quality, smart-city and research deployments) and get flat sensor rows — observations, things with coordinates, datastreams with units — plus raw JSON. @iot.nextLink paging, OData filters. Pay per row.", "version": "0.1", "x-build-id": "015k7g0YxD1D5Qyfv" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/datamule~sensorthings-api-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-datamule-sensorthings-api-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/datamule~sensorthings-api-extractor/runs": { "post": { "operationId": "runs-sync-datamule-sensorthings-api-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/datamule~sensorthings-api-extractor/run-sync": { "post": { "operationId": "run-sync-datamule-sensorthings-api-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "sources" ], "properties": { "sources": { "title": "SensorThings API URL(s)", "type": "array", "description": "One or more OGC SensorThings API v1.1 base URLs — the endpoint of ANY conforming instance (FROST-Server, GOST, SensorUp, …). A bare base URL is fine; the actor appends the entity-set path (/Things, /Datastreams, /Observations) itself, and a URL ending in an entity set is normalized back to the base. Examples: https://iot.hamburg.de/v1.1 (City of Hamburg), https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1 (Fraunhofer IOSB air quality), https://sensorthings.imk-ifu.kit.edu/v1.1 (KIT TERENO). One actor works against every OGC-compliant SensorThings deployment on Earth.", "items": { "type": "string" } }, "mode": { "title": "Mode", "enum": [ "observations", "things", "datastreams", "discovery" ], "type": "string", "description": "Which entity to harvest. observations (default) returns sensor readings (result + timestamps); things returns the monitored assets with their locations/coordinates; datastreams returns the measurement series (unit of measurement, observed area, phenomenon time) linking a Thing to a Sensor and ObservedProperty; discovery returns one fingerprint row per instance (available entity sets + declared conformance classes + server settings).", "default": "observations" }, "datastreamId": { "title": "Datastream id (observations mode)", "type": "string", "description": "Optional. In observations mode, scope the harvest to a single datastream by fetching /Datastreams({id})/Observations instead of the whole /Observations collection. Get datastream ids by running the actor once in datastreams mode. Leave empty to harvest all observations on the instance." }, "$filter": { "title": "OData $filter", "type": "string", "description": "Optional OData filter passed through verbatim to the SensorThings query, e.g. result gt 0, phenomenonTime gt 2024-01-01T00:00:00Z, name eq 'PM10', or startswith(name,'Air'). Leave empty for no filter. See the OGC SensorThings API v1.1 spec for the full $filter grammar." }, "$expand": { "title": "OData $expand", "type": "string", "description": "Optional OData expand passed through verbatim, e.g. Datastream,FeatureOfInterest on observations, or Sensor,ObservedProperty,Thing on datastreams, to inline related entities. In things mode this defaults to Locations (so each Thing carries its coordinates) unless you override it here." }, "$select": { "title": "OData $select", "type": "string", "description": "Optional OData select passed through verbatim to narrow the fields the server returns, e.g. @iot.id,result,phenomenonTime. Leave empty to return the full entity (the actor always keeps the raw entity in the _raw column)." }, "$orderby": { "title": "OData $orderby", "type": "string", "description": "Optional OData orderby passed through verbatim, e.g. phenomenonTime desc to get the newest observations first. Leave empty for the server default order." }, "maxRecords": { "title": "Max records (global cap)", "minimum": 1, "type": "integer", "description": "A GLOBAL cap on the total number of rows to harvest across all sources and all @iot.nextLink pages (each emitted row is one billable event). Default 1000. Increase to auto-paginate through a large collection; observations collections can be very large, so this is your cost control.", "default": 1000 }, "bearer": { "title": "Bearer token", "type": "string", "description": "Optional bearer token for gated SensorThings deployments (sent as Authorization: Bearer …). Not required for public instances — anonymous read is the norm for the protocol. Never logged." }, "extraHeaders": { "title": "Extra request headers", "type": "object", "description": "Optional extra HTTP headers as a JSON object, e.g. {\"x-api-key\": \"...\"} for gated servers. Not required for public instances. Header values are never logged." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```