# OTel Instrumentation Coverage Auditor (`michiganpublictech/otel-coverage-auditor`) Actor Find what isn't instrumented: dark dependencies, broken context propagation, uninstrumented time, missing resource metadata and log correlation gaps across a fleet of OpenTelemetry traces, with a prioritized fix plan. - **URL**: https://apify.com/michiganpublictech/otel-coverage-auditor.md - **Developed by:** [Justin Hagerty](https://apify.com/michiganpublictech) (community) - **Categories:** Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $750.00 / 1,000 trace set auditeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## OTel Instrumentation Coverage Auditor **Give it a batch of OpenTelemetry traces (and, optionally, logs and metrics). It tells you what *isn't* instrumented: services that never emit spans, hops where trace context gets lost, time inside requests that no span accounts for, missing resource metadata, and logs that can't be joined to traces. You also get a coverage score per service and a ranked list of fixes.** Most trace tools help you work out why one request was slow. This Actor answers a different question: **where is the trace data itself incomplete?** A trace can look healthy and still be missing half of what happened. A downstream service may have no SDK. A gateway may strip `traceparent`. One pod may run without the JDBC instrumentation. A Kafka consumer may start a new trace for every message. None of these raise errors. They leave gaps in the data that you only notice during an incident. Coverage is a property of the whole fleet, so the Actor analyses **many traces together**. It runs **55 rules** (`COV-*`) across topology, propagation, gaps, metadata, scopes, span semantics and logs/metrics correlation. It then builds a dependency graph (JSON plus Mermaid), scores every service from 0 to 100, and turns the findings into an **instrumentation improvement plan**. Each action in the plan has an effort level, an impact level and hints for the languages your services actually run. > **Results are suspected gaps, not proof.** A missing span can't prove that an operation happened. Sampling, export failures or an incomplete sample can look the same as missing instrumentation. Every finding is phrased as *suspected* and carries a `confidence` value. Verify before acting. *** ### Why use it - **Find dark dependencies.** These are internal services your code calls that never emit a span. The Actor separates them from databases, caches, brokers and SaaS APIs (`db.system`, `messaging.system`, known SaaS hostnames such as `api.stripe.com`), which aren't expected to emit spans. - **Find broken context propagation.** When a `CLIENT` span has no `SERVER` child in its trace, the Actor looks for a root `SERVER` span of the target service that started in another trace inside the client's time window, with the same route and method. That pins down the exact hop where context is lost. - **Measure uninstrumented time.** Child spans are merged as an *interval union*, so concurrent children aren't double-counted. Handlers whose children cover less than a threshold share of their duration are reported. So are handlers that make no calls at all in a service that is known to call databases or other services elsewhere. - **Catch inconsistent rollouts.** These include routes where only some requests show DB spans at similar latency, and pods that lack a scope their siblings have (for example `io.opentelemetry.jdbc`). The evidence is broken down by `service.version`. - **Fix metadata that breaks every dashboard.** These include `unknown_service:*`, missing `service.version` / `deployment.environment.name` / `service.instance.id`, inconsistent infra attributes, outdated or mixed SDK versions, and name collisions. - **Join logs and metrics to traces.** The Actor finds log records without trace context, trace IDs in logs that aren't in the trace set, and services that appear only in logs or metrics. It also finds services whose client metrics show database calls while no DB spans exist. - **Catch coverage regressions.** Pass a previous run as a `baseline` (or its dataset ID) to get per-service score deltas, services that disappeared and findings that are new since then. Plan actions that touch regressed services are ranked higher. - **Works with service meshes.** Envoy / Istio / Linkerd sidecar spans are recognised and collapsed, so proxies don't show up as badly instrumented "services" and calls through a mesh still pair up. - **Know what to do next.** The improvement plan is ordered by `impact × severity × confidence / effort`. Each action includes concrete steps such as `OTEL_PROPAGATORS=tracecontext,baggage`, enabling Java agent JDBC, adding `opentelemetry-instrumentation-psycopg`, setting `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES`, or adding span links for messaging. ### Who it's for - **Platform and observability teams** rolling out OpenTelemetry across many services, who need to know which services and hops still need work. - **SREs** after an incident where a trace "stopped" somewhere, who want to find every place that can happen. - **Consultancies** running observability-maturity assessments that need a repeatable, explainable report and score. - **CI pipelines** that export traces from a staging test run and fail the build when coverage regresses (`failOnSeverity`, SARIF output). ### How it works 1. **Parse.** Every input is normalised to one span model (lowercase hex IDs, nanosecond timestamps, resource, scope, kind, status, links). The formats are OTLP/JSON (hex *or* base64 IDs, legacy `instrumentationLibrarySpans`, Tempo `batches`), Jaeger JSON, Zipkin v2 (including `shared` client/server spans) and a simplified span list. Formats can be mixed in one trace set. Large exports can be read from an Apify dataset or key-value store record. Service-mesh proxy spans (Envoy / Istio / Linkerd, recognised by `component=proxy`, `upstream_cluster`, `istio.*`, `node_id=sidecar~…` and similar) are collapsed: their children are re-parented to the calling application span. 2. **Resolve peers.** Each outbound span (`CLIENT`/`PRODUCER`, or any span with `db.system`) is mapped to what it calls. Sources are tried in this order: the paired child span in the same trace, then `db.system` / `messaging.system`, then `peer.service`, then a host index learned from paired calls and `SERVER` span attributes, then the DNS label (`inventory.shop.svc.cluster.local` → `inventory`), then your expected services. Anything left over is classified by host: internal (no dot, `*.svc.cluster.local`, `*.internal`, private IPs and similar), SaaS (bundled suffix list) or external. 3. **Derive.** The Actor builds the dependency graph and dark dependencies. It matches orphans, roots and propagation breaks (one-to-one, closest start first, compatible route and method). It also computes gap groups (interval union per span), per-route downstream coverage by type (`db` / `remote` / `messaging`), per-instance scope sets, propagator hints, correlation IDs shared across traces, and the logs/metrics joins. 4. **Run the rules.** All 55 rules are pure functions. They only look at the telemetry, never at the network, so the same input always gives the same output. 5. **Score and plan.** Each service gets four sub-scores and a combined score (see [Scoring](#scoring)). The fleet gets one score, and findings are grouped into actions in the improvement plan. ### Rule categories | Category | Rules | Examples | |---|---|---| | Topology & discovery | `COV-TOPO-001…008` | Uninstrumented service named by `peer.service` (high). Internal host never seen as a service. Expected service never observed. Expected/unexpected edges from your service map. Outbound spans without any peer attributes. Isolated services. | | Propagation | `COV-PROP-001…012` | Orphan spans. Traces without a root. **Matched propagation breaks** (timing plus route/method, or a parent ID found in another trace). Called services starting new traces. Mixed propagators (b3 / tracecontext / jaeger / xray / datadog headers, 64-bit IDs). Messaging consumers without parent or link. Traces that start with a `CLIENT` span. Same request or message ID in several unlinked traces. `SERVER → SERVER` with no client span. Partial traces. Inconsistent `tracestate` sampling. | | Gaps | `COV-GAP-001…006` | Children cover less than `gapRatio` of a span. Opaque long handlers. Inconsistent downstream coverage per route. Instances instrumented differently. No `INTERNAL` spans in long handlers. Child spans that outlive their parent. | | Resource metadata | `COV-META-001…010` | `unknown_service:*`, missing `service.version` / environment / `service.instance.id`, inconsistent `host.*` / `k8s.*` / `cloud.*` attributes, missing `telemetry.sdk.*`, outdated SDK (approximate table), mixed SDK versions, one name used by several languages, deprecated `deployment.environment`. | | Instrumentation scopes | `COV-SCOPE-001…003` | Spans without a scope name. Services with manual spans only (no auto-instrumentation library). The same library at several versions. | | Span semantics | `COV-KIND-001…003`, `COV-STATUS-001…002` | Spans carrying HTTP/RPC attributes in a service that emits no `SERVER`/`CLIENT` spans. `SERVER` spans without HTTP/RPC/messaging attributes. DB spans that aren't `CLIENT`. Errors without `ERROR` status. HTTP server spans without `http.route`. | | Logs/metrics correlation | `COV-CORR-001…009` | Logs without trace context. Trace IDs only in MDC attributes. Log trace IDs missing from the trace set. Services in logs or metrics only. Server or DB client metrics without matching spans. Resource attributes that differ between signals. Log span IDs not in the trace. | | Data quality | `COV-DATA-001…002` | Small samples, duplicate span IDs. | Rule IDs are stable and never reused. See [RULES.md](RULES.md) for the full catalogue, with severity, rationale, recommendations and references. ### Scoring Each finding is attributed to the services in `evidence.services` and counted in one of four **dimensions**: | Dimension | Categories | Weight | |---|---|---| | `propagation` | topology, propagation | 35% | | `depth` | gaps, span semantics | 30% | | `metadata` | metadata, scopes | 20% | | `correlation` | logs/metrics correlation | 15% (dropped and the rest re-normalised when no logs or metrics were supplied) | A dimension starts at 100 and loses `points × confidence` per finding: high = 30, medium = 15, low = 6, info = 0. Repeated findings of the same severity decay by ×0.6 each, so twenty small issues can't outweigh one large one. The **service score** is the weighted mean of its dimensions. The **fleet coverage score** (the report's `score`) is the span-weighted mean of the service scores, minus fleet-wide findings such as an expected service that never appeared. `risk_level` is derived from the score in the same way as in all our Actors. Scores are deterministic, so you can track them over time and gate CI on them. ### How to use #### In Apify Console 1. Paste your traces into **Traces**. Start from the prefilled demo, or paste an export from your Collector's `file` exporter, the Jaeger UI ("Download JSON") or the Zipkin API. 2. Optionally list your **Expected services**, or provide an **Expected service map**. 3. Optionally add **Logs** and/or **Metrics** (OTLP/JSON) to run the correlation checks. 4. Click **Start**. The **Output** tab has an *Overview* view, a *Findings* view (one row per finding), a *Services* view, a *Topology & gaps* view and an *Improvement plan* view. The key-value store holds `OUTPUT`, `REPORT.md` and `REPORT.sarif`. #### Via the API ```bash jq -n --slurpfile t traces.json --slurpfile l logs.json \ '{traceSetName: "shop-prod", traces: $t, logs: $l, expectedServices: ["frontend","checkout","inventory"]}' | curl -s -X POST -H "Content-Type: application/json" -d @- \ "https://api.apify.com/v2/acts/michiganpublictech~otel-coverage-auditor/run-sync-get-dataset-items?token=$APIFY_TOKEN" | jq '.[0] | {score, summary, plan: [.data.improvement_plan[] | {rank, title, services}]}' ``` #### Python (apify-client) ```python import json from apify_client import ApifyClient client = ApifyClient("") run = client.actor("michiganpublictech/otel-coverage-auditor").call( run_input={ "traceSetName": "staging-load-test", "traces": [json.load(open("otlp-traces.json"))], # wrap one export document in a list "serviceMap": {"frontend": ["checkout", "cart"], "checkout": ["inventory", "postgresql"]}, "failOnSeverity": "high", } ) report = next(client.dataset(run["defaultDatasetId"]).iterate_items()) print("fleet coverage", report["score"]) for s in report["data"]["services"]: print(f"{s['service']:20} {s['score']:3} {s['breakdown']}") print(report["data"]["dependency_graph"]["mermaid"]) ``` #### Getting traces out of your stack - **OpenTelemetry Collector:** add a `file` exporter (`format: json`) to the traces pipeline for 10–15 minutes. The output is JSON Lines of `ExportTraceServiceRequest`, and you can pass it as a string inside `traces` (`"traces": [""]`). - **Jaeger:** use the UI's *Download JSON* or `GET /api/traces?service=…&limit=…`. The `{"data": [...]}` document works as-is. - **Zipkin:** `GET /api/v2/traces?limit=…` returns a list of span lists. Pass it as-is. - **Anything else:** map spans to the simplified format below. #### Large exports (Apify storage) The Actor input is limited to a few MB. For bigger exports, store the traces in Apify storage and reference them: - **Dataset** (`traceDatasetId`): one span (simplified or Zipkin v2) or one export document (OTLP/JSON, Jaeger JSON) per item. Up to 200,000 items are read. - **Key-value store record** (`traceRecordKey`, plus `traceKeyValueStoreId` unless the record is in the run's default store): one OTLP/JSON or Jaeger document, a span list, or JSON Lines text straight from the Collector `file` exporter. Storage input is added to the `traces` trace set. Inside `traceSets`, an entry can use `datasetId`, or `recordKey` plus optional `keyValueStoreId`, instead of inline `traces`. A dataset or record that can't be read becomes a free `failed` item, and the run continues. Storages are opened by **ID** (never by name, which would create a missing storage). `traceDatasetId`, `traceKeyValueStoreId` and `baselineDatasetId` use Apify's storage picker and request read access only, so they also work when the Actor runs with limited permissions. Storage IDs inside `traceSets` entries are opened with the run's token and must be accessible to it. Each set is still capped at 100,000 spans. ```python from apify_client import ApifyClient client = ApifyClient("") store = client.key_value_stores().get_or_create(name="otel-exports") client.key_value_store(store["id"]).set_record( "shop-prod", open("traces.jsonl").read(), content_type="text/plain", # Collector file exporter output ) run = client.actor("michiganpublictech/otel-coverage-auditor").call( run_input={"traceSetName": "shop-prod", "traceRecordKey": "shop-prod", "traceKeyValueStoreId": store["id"]} ) ``` ### Input | Field | Type | Description | |---|---|---| | `traces` | array | The trace set to audit. Either a list of **spans** (simplified or Zipkin v2) or a list of **documents** (OTLP/JSON `ExportTraceServiceRequest`, Jaeger JSON). Items may also be JSON / JSON Lines strings. | | `traceSetName` | string | Report target name for `traces` (default `traces`). | | `traceSets` | array | More independent sets in one run: `[{name, traces, logs?, metrics?, expectedServices?, serviceMap?}]`. Each set gets its own report and is billed separately. An entry can read its traces from storage with `datasetId`, or `recordKey` + optional `keyValueStoreId`. | | `traceDatasetId` | string | ID of an Apify dataset holding spans or export documents (see [Large exports](#large-exports-apify-storage)). | | `traceRecordKey` | string | Key of a key-value store record holding an export (JSON, or JSON Lines text). | | `traceKeyValueStoreId` | string | ID of the key-value store holding `traceRecordKey` (default: the run's default store). | | `expectedServices` | array | `service.name` values you expect. Inside `traceSets` this field can also be a `{service: [deps]}` map. | | `serviceMap` | object | `{service: [dependencies]}`. Keys become expected services. Edges that never appear are reported. Dependencies may be services, `db.system` / `messaging.system` names or host labels. | | `logs` | array | OTLP/JSON `ExportLogsServiceRequest` documents or simplified records `{service, traceId, spanId, time}`. Log bodies are never copied to the output. | | `metrics` | array | OTLP/JSON `ExportMetricsServiceRequest` documents or `{service, metrics: [names], resource?}`. Only resources and metric names are used. | | `minGapMs` | integer | Minimum uninstrumented time worth reporting (default **100** ms). | | `gapRatio` | number | Report a span when its children's interval union covers less than this share of it (default **0.5**). | | `clockSkewMs` | integer | Clock skew allowed when matching a call to a server span that started a new trace (default **100** ms, 0–60,000). Raise it if your hosts' clocks drift. | | `baseline` | array | A previous run's trace-set dataset items, or compact summaries `[{traceSet?, services: {name: score}, fleetScore?}]`. Enables `data.regressions`. | | `baselineDatasetId` | string | ID of a previous run's default dataset to use as the baseline. If it can't be read, the run continues without it (with a note). | | `perServiceItems` | boolean | Also push one free dataset item per service (default `false`). | | `failOnSeverity` | string | CI gate: `none` (default) / `critical` / `high` / `medium` / `low`. | | `disabledRules` | array | Rule IDs to skip (plans with custom rules). | | `licenseKey` | string (secret) | Optional enterprise or prepaid key. | #### Input formats with examples **Simplified span list.** Times are epoch milliseconds or ISO-8601. Use `durationMs` or `endTime`. `kind` defaults to `INTERNAL`. Common aliases such as `traceID` and `serviceName` are accepted. ```json { "traces": [ {"traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60003", "parentSpanId": "a1b2c3d4e5f60002", "service": "checkout", "name": "POST /checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.007Z", "durationMs": 395, "attributes": {"http.request.method": "POST", "http.route": "/checkout"}, "resource": {"service.version": "1.5.0", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0"}, "scope": {"name": "io.opentelemetry.spring-webmvc-6.0"}} ] } ``` **OTLP/JSON.** Wrap the document in a list. IDs can be hex (as the OTLP/JSON spec requires) or base64 (as protobuf `MessageToJson` produces). ```json {"traces": [{"resourceSpans": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "cart"}}]}, "scopeSpans": [{"scope": {"name": "OpenTelemetry.Instrumentation.AspNetCore"}, "spans": [ {"traceId": "5b8efff798038103d269b633813fc60c", "spanId": "eee19b7ec3c1b174", "name": "GET /cart/{id}", "kind": 2, "startTimeUnixNano": "1790000000000000000", "endTimeUnixNano": "1790000000032000000"}]}]}]}]} ``` **Jaeger and Zipkin.** Paste the API output as-is. For Jaeger, `span.kind`, `otel.library.name` / `otel.scope.name` and `error` / `otel.status_code` tags are understood, and `FOLLOWS_FROM` references become links. For Zipkin, `remoteEndpoint.serviceName` becomes `peer.service` and `shared` server spans are separated from their client half. **Logs (simplified).** ```json {"logs": [{"service": "checkout", "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60003", "time": 1790000000000}]} ``` ### Output Each trace set produces one dataset item in the shared `Report` envelope (`product`, `target`, `score`, `risk_level`, `summary`, `findings[]`, `data{}`, `errors[]`, `disclaimer`), so `REPORT.md`, SARIF and the CI gate work like in every other Actor in this family. The product payload is in `data`: | Field | Content | |---|---| | `fleet_coverage_score` | The fleet score (same value as `score`). | | `stats` | Traces, spans, services, resources, detected formats, time range, root and orphan span counts, collapsed mesh-proxy spans. | | `services[]` | Per service: `score`, `breakdown{propagation,depth,metadata,correlation}`, spans, traces, span kinds, instances, versions, languages, SDK versions, `auto_instrumented`, entry / root-entry / orphan / outbound span counts, `calls`, `called_by`, finding counts. Log correlation share is included when logs were supplied. Sorted worst first. | | `dependency_graph` | `nodes[]` (type `service` / `database` / `cache` / `messaging` / `saas` / `external` / `internal-unknown`, plus a `suspected_dark` flag), `edges[]` (calls, paired calls, status `traced` / `partially traced` / `client-only` / `suspected uninstrumented`) and `mermaid` text. | | `dark_dependencies[]` | Dependency, how it was identified, callers with call counts, hosts, operations, example trace ID. | | `propagation_breaks[]` | Matched client → server pairs across traces: both trace IDs, start offset, and what the match was based on. | | `gaps[]` | Per service and operation: `low-coverage` / `opaque`, occurrences, median child coverage, median uninstrumented ms, largest contiguous gap. | | `route_coverage[]` | Routes with inconsistent downstream coverage, by downstream type and broken down by version. | | `scope_inventory` | Per service: instrumentation scopes, versions, span counts, whether each is auto-instrumentation. | | `correlation` | Log and metric join statistics, or `null` when no logs or metrics were supplied. | | `improvement_plan[]` | Ranked actions (see below). With a baseline, each action lists `regressed_services`, and actions touching them get 1.5× priority. | | `regressions` | Only with a baseline (otherwise `null`): baseline and current fleet score and delta, per-service `baseline_score` / `score` / `delta` / `status` (`regressed` at −5 points or worse, `improved`, `stable`), `regressed_services`, `improved_services`, `new_services`, `missing_services`, and `new_findings` / `resolved_findings` (when the baseline carried findings). | | `data_quality[]` | Notes about the input: mixed formats, base64 IDs, missing timestamps, duplicates, small sample, truncation. | | `thresholds`, `billing`, `notes` | Effective thresholds (`min_gap_ms`, `gap_ratio`, `clock_skew_ms`), billed units, run notes. | Example finding: ```json { "rule_id": "COV-PROP-003", "severity": "high", "kind": "observation", "category": "propagation", "title": "Suspected propagation break checkout → inventory (6 matched call(s))", "location": "edge:checkout->inventory", "confidence": 0.9, "evidence": {"services": ["checkout", "inventory"], "matched_calls": 6, "matched_on_route": 6, "examples": [{"client_trace_id": "1234…", "server_trace_id": "9f0e…", "start_offset_ms": 3.0, "matched_on": "timing + route/method"}]} } ``` Example Mermaid graph from the demo fleet. Dashed edges are untraced or partially traced, and dashed red nodes are suspected dark dependencies. ```mermaid graph LR n3["checkout"] n5["fraud-check (no spans?)"]:::dark n6["frontend"] n7["inventory"] n10[("postgresql:orders-db.shop.svc.cluster.local")] n6 -->|"18"| n3 n3 -.->|"18 suspected uninstrumented"| n5 n3 -.->|"18 partially traced"| n7 n3 -->|"12"| n10 classDef dark stroke-dasharray: 5 5,stroke:#d9534f ``` #### Improvement plan example ```json [ {"rank": 1, "action": "set-service-identity", "title": "Name every service and describe its deployment", "effort": "low", "impact": "high", "services": ["recommendation", "unknown_service:java"], "rules": ["COV-META-001", "COV-META-002"], "hints": {"generic": "Set OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES=service.version=,deployment.environment.name= …", "java": "-Dotel.service.name=checkout -Dotel.resource.attributes=service.version=${GIT_SHA} …", "python": "Set the env vars before `opentelemetry-instrument`, or Resource.create({...}) in code."}}, {"rank": 2, "action": "fix-propagation-boundaries", "title": "Repair broken propagation boundaries", "effort": "medium", "impact": "high", "services": ["checkout", "email-worker", "inventory"]}, {"rank": 3, "action": "data-instrumentation", "title": "Instrument database, cache and messaging clients", "effort": "low", "impact": "high", "services": ["inventory", "recommendation"], "hints": {"python": "pip install opentelemetry-instrumentation-psycopg (or -psycopg2, -sqlalchemy, -asyncpg, -redis, -pymongo)."}} ] ``` Per-language hints cover `java`, `python`, `nodejs`, `go` and `dotnet`. Only the languages detected in the affected services (`telemetry.sdk.language`) are included. If no language is known, all hints are included. ### Pricing This Actor uses **pay-per-event** pricing. You pay only for results, and Apify platform usage is included. | Event | Price | When it is charged | |---|---|---| | `coverage-audited` | **$0.75** | Once per audited **trace set** of up to **10,000 spans**. Larger sets are charged one event per started 10,000 spans (`ceil(spans / 10,000)`), up to 100,000 spans per set. | | `correlation-check` | **$0.25** | Once per trace set, **only** when you supplied logs and/or metrics and the correlation actually ran. | - Trace sets that can't be parsed (no spans found) are **free**, and per-service items are free. - Before each set, the Actor checks that your remaining budget ("maximum cost per run") covers the **combined** cost of all of that set's units (plus the correlation check, when requested). If it doesn't, the run stops before doing unpaid work. If the budget covers the audit but not the correlation check, the audit runs without correlation and you aren't charged for it. - Examples: one 4,000-span trace set = **$0.75**. The same set with logs = **$1.00**. Three environments of 25,000 spans each, with metrics = 3 × (3 × $0.75 + $0.25) = **$7.50**. ### Plans and licensing Apify Store users get the full product on pay-per-event pricing, including SARIF export and the CI gate. License keys (`licenseKey`) unlock more: **Pro** lets you disable rules (`disabledRules`) and audit more trace sets per run, **Team** adds bulk runs, and **Enterprise** adds white-label reports. Prepaid keys skip pay-per-event charges. Local runs without a key use the Community plan, which allows up to 3 trace sets per run. Missing features never fail a run; they are skipped with a note. See [docs/licensing.md](../../docs/licensing.md). ### Limitations - **Inference, not proof.** Every gap is a *suspected* coverage gap. Head sampling, SDK queue overflows, collector drops and partial exports look the same as missing instrumentation. The `confidence` value and the `data_quality` notes help you judge each one. - **Sample size matters.** Coverage is a fleet property. Fewer than about 10 traces gives weak conclusions (`COV-DATA-001`). A few hundred traces covering your main routes works well. - **Timing-based matching depends on clocks.** Propagation-break matching allows 100 ms of clock skew by default (`clockSkewMs`) and requires compatible route and method when both sides record them. Very busy routes without route attributes can produce ambiguous matches, which get lower confidence. - **Host classification is heuristic.** "Internal" hosts are recognised by naming (no dot, `*.svc.cluster.local`, `*.internal`, private IPs, …). The SaaS list is curated. Set `peer.service` or `db.system` on client spans, or provide `expectedServices`, to remove ambiguity. - **The SDK version table is approximate** (a snapshot from this release). It only flags clearly old SDKs. - **Zipkin and Jaeger exports carry less metadata** (no scopes, partial resources), so metadata and scope checks are less precise for them. - **Input size.** Inline input is limited to a few MB; use a dataset or key-value store record for larger exports. Each set is capped at 100,000 spans (and 200,000 dataset items); whole traces beyond that are skipped, with a note. Logs and metrics must still be inline. - **Heuristics are tuned on synthetic fleets.** Regression fixtures cover async messaging (child and linked consumers), cache-hit routes, batch jobs and Istio sidecars, but real exports can still produce false positives. Use `disabledRules` and report patterns that look wrong. - **No network access.** The Actor never fetches traces from your backend. You send it exported data (inline or in Apify storage). ### FAQ **Does any of my telemetry leave Apify?** No. The Actor makes no network calls outside Apify (it only reads the Apify datasets or records you point it to). Log bodies and attribute values are never copied to the output. Evidence contains service names, span names, hosts, routes, IDs and counts. **A database shows up as a dependency without spans. Is that a problem?** No. Databases, caches, brokers (`db.system`, `messaging.system`) and SaaS or external hosts are classified as managed or external and are never reported as dark dependencies. Only internal hosts and `peer.service` names that never emit spans are reported. **Why is a dark dependency reported when the service *is* instrumented?** Usually because its `service.name` doesn't match the name or host the callers use, or because it exports to a different backend. Add it to `expectedServices`, or set `peer.service` on the client spans to the callee's `service.name`. **Why does `COV-GAP-001` fire on an endpoint that is pure CPU work?** Uncovered time can be CPU work. The finding means "this time isn't attributed". Add manual `INTERNAL` spans if the time matters, or raise `minGapMs` / lower `gapRatio`. **How do I run it in CI?** Export traces from an integration-test run (Collector `file` exporter), call the Actor with `failOnSeverity: "high"`, and upload `REPORT.sarif` to GitHub code scanning. To catch coverage regressions, pass the previous run's dataset as `baselineDatasetId` and read `data.regressions`. **How do I silence a rule?** Use `disabledRules` (plans with custom rules). Rule IDs are stable. ### Changelog See [CHANGELOG.md](CHANGELOG.md). # Changelog This Actor's version history is a separate document: https://apify.com/michiganpublictech/otel-coverage-auditor/changelog.md # Actor input Schema ## `traces` (type: `array`): Spans or export documents to audit as one trace set. Accepts a list of spans (simplified format or Zipkin v2), or a list of documents: OTLP/JSON ExportTraceServiceRequest ({"resourceSpans": …}, hex or base64 ids), Jaeger JSON ({"data": \[...]}) — wrap a single document in \[ ]. Strings holding JSON / JSON Lines (Collector file exporter output) are accepted as list items. ## `traceSetName` (type: `string`): Name shown as the report target for the 'traces' field (e.g. 'shop-production 2026-10-01'). ## `traceSets` (type: `array`): Audit several independent trace sets in one run, each producing its own report and billed separately: \[{"name": "prod", "traces": \[...], "logs": \[...], "metrics": \[...], "expectedServices": \[...] or {service: \[deps]}}]. Instead of (or as well as) inline 'traces', an entry can read its traces from storage with "datasetId", or "recordKey" plus optional "keyValueStoreId". ## `traceDatasetId` (type: `string`): Apify dataset whose items are spans (simplified or Zipkin v2) or whole export documents (OTLP/JSON, Jaeger JSON). Up to 200,000 items are read. ## `traceRecordKey` (type: `string`): Key of a key-value store record holding an export: OTLP/JSON, Jaeger JSON, a span list, or JSON Lines text (Collector file exporter). ## `traceKeyValueStoreId` (type: `string`): Key-value store holding 'Traces record key'. Leave empty to use this run's default store. ## `expectedServices` (type: `array`): service.name values you expect to see. Services never observed in any signal are reported, and calls to these names are treated as calls to your own (instrumentable) services. ## `serviceMap` (type: `object`): Optional {service: \[dependencies]} map, e.g. {"checkout": \["inventory", "postgresql"]}. Keys are added to the expected services; declared edges that never appear are reported. Dependencies can be services, db.system / messaging.system names or host labels. ## `logs` (type: `array`): ExportLogsServiceRequest documents ({"resourceLogs": …}) or simplified records ({"service", "traceId", "spanId", "time"}). Used to find log records without trace context, trace ids missing from the trace set, and services that log but don't trace. Log bodies are never copied to the output. ## `metrics` (type: `array`): ExportMetricsServiceRequest documents ({"resourceMetrics": …}) or simplified resources ({"service": "billing", "metrics": \["db.client.operation.duration"]}). Only resources and metric names are used (no values). ## `minGapMs` (type: `integer`): Spans shorter than this, and uncovered time below it, are never reported as gaps. ## `gapRatio` (type: `number`): A SERVER/CONSUMER/INTERNAL span is a suspected gap when the union of its children's intervals covers less than this share of its duration (0.01–1). ## `clockSkewMs` (type: `integer`): How far apart caller and callee clocks may be when matching a call to a server span that started a new trace (propagation breaks). Raise it if hosts' clocks drift; larger values allow more timing-only matches on busy routes. ## `baseline` (type: `array`): Trace-set dataset items from a previous run of this Actor (copied from its dataset), or compact summaries \[{"traceSet": "shop-production", "services": {"checkout": 72}, "fleetScore": 70}]. A single entry without a trace set name applies to every set. ## `baselineDatasetId` (type: `string`): A previous run's default dataset. Its trace-set items are used as the baseline (per-service items are ignored). If it can't be read, the run continues without a baseline. ## `perServiceItems` (type: `boolean`): Push an extra (free) dataset item per service with its score, breakdown and findings, in addition to the trace-set report. ## `failOnSeverity` (type: `string`): CI gate: mark the run as FAILED when any finding is at or above this severity. ## `disabledRules` (type: `array`): Rule IDs to skip, e.g. COV-META-004 (see RULES.md). Requires the custom_rules feature (Pro plan and above). ## `licenseKey` (type: `string`): Optional. Enterprise / prepaid license key. Store users don't need one. ## Actor input object example ```json { "traces": [ { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60001", "name": "POST /api/checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.000Z", "durationMs": 420, "attributes": { "http.request.method": "POST", "http.route": "/api/checkout", "http.response.status_code": 200 }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0" }, "scope": "@opentelemetry/instrumentation-http" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60002", "parentSpanId": "a1b2c3d4e5f60001", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.005Z", "durationMs": 400, "attributes": { "http.request.method": "POST", "server.address": "checkout.shop.svc.cluster.local", "url.full": "http://checkout.shop.svc.cluster.local:8080/checkout" }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0" }, "scope": "@opentelemetry/instrumentation-http" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60003", "parentSpanId": "a1b2c3d4e5f60002", "name": "POST /checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.007Z", "durationMs": 395, "attributes": { "http.request.method": "POST", "http.route": "/checkout", "http.response.status_code": 200 }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.spring-webmvc-6.0" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60004", "parentSpanId": "a1b2c3d4e5f60003", "name": "SELECT orders", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.010Z", "durationMs": 60, "attributes": { "db.system.name": "postgresql", "db.operation.name": "SELECT", "server.address": "orders-db.shop.svc.cluster.local" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.jdbc" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60005", "parentSpanId": "a1b2c3d4e5f60003", "name": "GET", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.072Z", "durationMs": 50, "attributes": { "http.request.method": "GET", "server.address": "inventory.shop.svc.cluster.local", "url.full": "http://inventory.shop.svc.cluster.local:8080/stock/1042" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60006", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.124Z", "durationMs": 80, "attributes": { "http.request.method": "POST", "server.address": "fraud-check.shop.svc.cluster.local", "url.full": "http://fraud-check.shop.svc.cluster.local:8080/score" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60007", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.206Z", "durationMs": 120, "attributes": { "http.request.method": "POST", "server.address": "api.stripe.com", "url.full": "https://api.stripe.com/v1/payment_intents" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "8f2a1c9d3e4b5a6f7081928374655647", "spanId": "b1b2c3d4e5f60001", "name": "GET /stock/{sku}", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.075Z", "durationMs": 44, "attributes": { "http.request.method": "GET", "http.route": "/stock/{sku}", "url.path": "/stock/1042", "http.response.status_code": 200 }, "resource": { "service.name": "inventory", "deployment.environment.name": "production", "telemetry.sdk.language": "python", "telemetry.sdk.version": "1.16.0" }, "scope": "opentelemetry.instrumentation.fastapi" } ], "traceSetName": "shop-production", "expectedServices": [ "frontend", "checkout", "inventory", "shipping" ], "minGapMs": 100, "gapRatio": 0.5, "clockSkewMs": 100, "perServiceItems": false, "failOnSeverity": "none" } ``` # Actor output Schema ## `results` (type: `string`): Coverage report per trace set: fleet score, services table, dependency graph, dark dependencies, propagation breaks, gaps, improvement plan and findings. ## `summary` (type: `string`): Plan, per-target scores and billing summary for the run (OUTPUT record). ## `reportMarkdown` (type: `string`): Human-readable report of all findings (REPORT.md). ## `reportSarif` (type: `string`): SARIF 2.1.0 report for GitHub code scanning and other CI tools (REPORT.sarif). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "traces": [ { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60001", "name": "POST /api/checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.000Z", "durationMs": 420, "attributes": { "http.request.method": "POST", "http.route": "/api/checkout", "http.response.status_code": 200 }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0" }, "scope": "@opentelemetry/instrumentation-http" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60002", "parentSpanId": "a1b2c3d4e5f60001", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.005Z", "durationMs": 400, "attributes": { "http.request.method": "POST", "server.address": "checkout.shop.svc.cluster.local", "url.full": "http://checkout.shop.svc.cluster.local:8080/checkout" }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0" }, "scope": "@opentelemetry/instrumentation-http" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60003", "parentSpanId": "a1b2c3d4e5f60002", "name": "POST /checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.007Z", "durationMs": 395, "attributes": { "http.request.method": "POST", "http.route": "/checkout", "http.response.status_code": 200 }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.spring-webmvc-6.0" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60004", "parentSpanId": "a1b2c3d4e5f60003", "name": "SELECT orders", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.010Z", "durationMs": 60, "attributes": { "db.system.name": "postgresql", "db.operation.name": "SELECT", "server.address": "orders-db.shop.svc.cluster.local" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.jdbc" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60005", "parentSpanId": "a1b2c3d4e5f60003", "name": "GET", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.072Z", "durationMs": 50, "attributes": { "http.request.method": "GET", "server.address": "inventory.shop.svc.cluster.local", "url.full": "http://inventory.shop.svc.cluster.local:8080/stock/1042" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60006", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.124Z", "durationMs": 80, "attributes": { "http.request.method": "POST", "server.address": "fraud-check.shop.svc.cluster.local", "url.full": "http://fraud-check.shop.svc.cluster.local:8080/score" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60007", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.206Z", "durationMs": 120, "attributes": { "http.request.method": "POST", "server.address": "api.stripe.com", "url.full": "https://api.stripe.com/v1/payment_intents" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "8f2a1c9d3e4b5a6f7081928374655647", "spanId": "b1b2c3d4e5f60001", "name": "GET /stock/{sku}", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.075Z", "durationMs": 44, "attributes": { "http.request.method": "GET", "http.route": "/stock/{sku}", "url.path": "/stock/1042", "http.response.status_code": 200 }, "resource": { "service.name": "inventory", "deployment.environment.name": "production", "telemetry.sdk.language": "python", "telemetry.sdk.version": "1.16.0" }, "scope": "opentelemetry.instrumentation.fastapi" } ], "traceSetName": "shop-production", "expectedServices": [ "frontend", "checkout", "inventory", "shipping" ] }; // Run the Actor and wait for it to finish const run = await client.actor("michiganpublictech/otel-coverage-auditor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "traces": [ { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60001", "name": "POST /api/checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.000Z", "durationMs": 420, "attributes": { "http.request.method": "POST", "http.route": "/api/checkout", "http.response.status_code": 200, }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0", }, "scope": "@opentelemetry/instrumentation-http", }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60002", "parentSpanId": "a1b2c3d4e5f60001", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.005Z", "durationMs": 400, "attributes": { "http.request.method": "POST", "server.address": "checkout.shop.svc.cluster.local", "url.full": "http://checkout.shop.svc.cluster.local:8080/checkout", }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0", }, "scope": "@opentelemetry/instrumentation-http", }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60003", "parentSpanId": "a1b2c3d4e5f60002", "name": "POST /checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.007Z", "durationMs": 395, "attributes": { "http.request.method": "POST", "http.route": "/checkout", "http.response.status_code": 200, }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0", }, "scope": "io.opentelemetry.spring-webmvc-6.0", }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60004", "parentSpanId": "a1b2c3d4e5f60003", "name": "SELECT orders", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.010Z", "durationMs": 60, "attributes": { "db.system.name": "postgresql", "db.operation.name": "SELECT", "server.address": "orders-db.shop.svc.cluster.local", }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0", }, "scope": "io.opentelemetry.jdbc", }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60005", "parentSpanId": "a1b2c3d4e5f60003", "name": "GET", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.072Z", "durationMs": 50, "attributes": { "http.request.method": "GET", "server.address": "inventory.shop.svc.cluster.local", "url.full": "http://inventory.shop.svc.cluster.local:8080/stock/1042", }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0", }, "scope": "io.opentelemetry.java-http-client", }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60006", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.124Z", "durationMs": 80, "attributes": { "http.request.method": "POST", "server.address": "fraud-check.shop.svc.cluster.local", "url.full": "http://fraud-check.shop.svc.cluster.local:8080/score", }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0", }, "scope": "io.opentelemetry.java-http-client", }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60007", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.206Z", "durationMs": 120, "attributes": { "http.request.method": "POST", "server.address": "api.stripe.com", "url.full": "https://api.stripe.com/v1/payment_intents", }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0", }, "scope": "io.opentelemetry.java-http-client", }, { "traceId": "8f2a1c9d3e4b5a6f7081928374655647", "spanId": "b1b2c3d4e5f60001", "name": "GET /stock/{sku}", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.075Z", "durationMs": 44, "attributes": { "http.request.method": "GET", "http.route": "/stock/{sku}", "url.path": "/stock/1042", "http.response.status_code": 200, }, "resource": { "service.name": "inventory", "deployment.environment.name": "production", "telemetry.sdk.language": "python", "telemetry.sdk.version": "1.16.0", }, "scope": "opentelemetry.instrumentation.fastapi", }, ], "traceSetName": "shop-production", "expectedServices": [ "frontend", "checkout", "inventory", "shipping", ], } # Run the Actor and wait for it to finish run = client.actor("michiganpublictech/otel-coverage-auditor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "traces": [ { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60001", "name": "POST /api/checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.000Z", "durationMs": 420, "attributes": { "http.request.method": "POST", "http.route": "/api/checkout", "http.response.status_code": 200 }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0" }, "scope": "@opentelemetry/instrumentation-http" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60002", "parentSpanId": "a1b2c3d4e5f60001", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.005Z", "durationMs": 400, "attributes": { "http.request.method": "POST", "server.address": "checkout.shop.svc.cluster.local", "url.full": "http://checkout.shop.svc.cluster.local:8080/checkout" }, "resource": { "service.name": "frontend", "service.version": "2.3.1", "deployment.environment.name": "production", "service.instance.id": "frontend-7d9f", "telemetry.sdk.language": "nodejs", "telemetry.sdk.version": "1.28.0" }, "scope": "@opentelemetry/instrumentation-http" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60003", "parentSpanId": "a1b2c3d4e5f60002", "name": "POST /checkout", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.007Z", "durationMs": 395, "attributes": { "http.request.method": "POST", "http.route": "/checkout", "http.response.status_code": 200 }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.spring-webmvc-6.0" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60004", "parentSpanId": "a1b2c3d4e5f60003", "name": "SELECT orders", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.010Z", "durationMs": 60, "attributes": { "db.system.name": "postgresql", "db.operation.name": "SELECT", "server.address": "orders-db.shop.svc.cluster.local" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.jdbc" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60005", "parentSpanId": "a1b2c3d4e5f60003", "name": "GET", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.072Z", "durationMs": 50, "attributes": { "http.request.method": "GET", "server.address": "inventory.shop.svc.cluster.local", "url.full": "http://inventory.shop.svc.cluster.local:8080/stock/1042" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60006", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.124Z", "durationMs": 80, "attributes": { "http.request.method": "POST", "server.address": "fraud-check.shop.svc.cluster.local", "url.full": "http://fraud-check.shop.svc.cluster.local:8080/score" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "a1b2c3d4e5f60007", "parentSpanId": "a1b2c3d4e5f60003", "name": "POST", "kind": "CLIENT", "startTime": "2026-10-01T12:00:00.206Z", "durationMs": 120, "attributes": { "http.request.method": "POST", "server.address": "api.stripe.com", "url.full": "https://api.stripe.com/v1/payment_intents" }, "resource": { "service.name": "checkout", "service.version": "1.5.0", "deployment.environment.name": "production", "service.instance.id": "checkout-5c2a", "telemetry.sdk.language": "java", "telemetry.sdk.version": "1.43.0" }, "scope": "io.opentelemetry.java-http-client" }, { "traceId": "8f2a1c9d3e4b5a6f7081928374655647", "spanId": "b1b2c3d4e5f60001", "name": "GET /stock/{sku}", "kind": "SERVER", "startTime": "2026-10-01T12:00:00.075Z", "durationMs": 44, "attributes": { "http.request.method": "GET", "http.route": "/stock/{sku}", "url.path": "/stock/1042", "http.response.status_code": 200 }, "resource": { "service.name": "inventory", "deployment.environment.name": "production", "telemetry.sdk.language": "python", "telemetry.sdk.version": "1.16.0" }, "scope": "opentelemetry.instrumentation.fastapi" } ], "traceSetName": "shop-production", "expectedServices": [ "frontend", "checkout", "inventory", "shipping" ] }' | apify call michiganpublictech/otel-coverage-auditor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,michiganpublictech/otel-coverage-auditor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Trfd5gLEQsCSAKQuv/builds/Gx0h4qqH8NmYZIxkT/openapi.json