# Distributed Trace Root Cause Analyzer (`michiganpublictech/trace-root-cause-analyzer`) Actor

Find why a request was slow or failed from distributed traces (OTLP, Jaeger, Zipkin): critical path, error origin, slow queries, N+1, retries, timeouts and dependency bottlenecks, ranked with evidence.

- **URL**: https://apify.com/michiganpublictech/trace-root-cause-analyzer.md
- **Developed by:** [Justin Hagerty](https://apify.com/michiganpublictech) (community)
- **Categories:** Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $500.00 / 1,000 trace investigateds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Distributed Trace Root Cause Analyzer

**Paste a distributed trace (OpenTelemetry OTLP/JSON, Jaeger, Zipkin or a simple JSON shape) and get back *why* the request was slow or failed: the critical path, where the error originated, and a ranked list of suspected root causes, each with a confidence score, the evidence behind it and what to do next.**

Trace UIs show you a waterfall of hundreds of spans. Working out from that waterfall which span actually made the request slow, which error caused the failure and which errors only propagated it, and whether you are looking at an N+1 loop, a retry storm, an exhausted connection pool or a cold start, takes an experienced engineer a while. This Actor does that triage automatically and deterministically, using the same reasoning an SRE would:

- **Critical-path analysis** that handles concurrent calls correctly, so you see the work that determined latency, not merely the longest spans.
- **Error origin vs. propagation**: it walks the error chain from the failed request down to the deepest failing span that explains the others.
- **56 trace detectors**: slow and N+1 queries, sequential calls that could run in parallel, fan-out explosions, retries and retry storms, timeouts and timeout-budget inversions, connection-pool waits, lock contention, cache misses and stampedes, cold starts, GC hints, consumer lag, throttling, gateway errors, cascading failures, external dependency slowness, clock skew and more.
- **Root-cause ranking** that combines critical-path contribution, error origin, deviation from healthy baseline traces and detector evidence into an explained score (`high` / `medium` / `low`).
- **Baseline comparison** (optional): per-operation latency regressions with robust statistics (median/p95), critical-path shifts, and new operations, services and errors compared with healthy traces.
- **Fleet summary** across many traces (optional, one extra item per run): which service or operation is most often the root cause or bottleneck, latency percentiles per operation, and the SLO breach rate.

No LLM and no calls to your systems. The analysis is a pure function of the spans you provide, so the same input always gives the same answer.

> ⚠️ **This is heuristic analysis of the spans you supplied, not proof.** Missing instrumentation, sampling, clock skew and partial exports can hide the real cause. Every suspect comes with its reasoning and evidence so you can verify it with logs, metrics and the systems involved.

***

### Who it's for

- **On-call engineers and SREs**: drop in the trace from the alert and get a first diagnosis in seconds, plus the next step to check.
- **Platform and observability teams**: run it on a batch of slow or failed traces from an incident to see the recurring cause, or put it in a pipeline after synthetic checks.
- **Developers reviewing performance**: catch N+1 queries, sequential awaits and chatty calls in traces from staging or load tests.
- **AI agents and automations**: the structured JSON output (`suspected_root_causes`, `critical_path`, `error_chain`) is designed to be consumed by other tools.

### How to use

#### In Apify Console

1. Open the Actor and paste your trace into **Traces** (a list: wrap a single payload in `[ … ]`). The prefilled example is a checkout request with an N+1 query, a slow query and a slow third-party call.
2. Optionally add **Baseline traces** (healthy traces of the same request), a **Latency SLO (ms)** or a **Focus service**.
3. Click **Start**. Results:
   - **Output → Overview / Root causes / Critical path / Findings** tabs (one row per trace).
   - Key-value store: **`INVESTIGATION.md`** (the full write-up with critical-path tables, Mermaid diagrams, waterfalls and error chains), `REPORT.md` (findings), `REPORT.sarif` (on plans with SARIF) and `OUTPUT` (run summary).

#### Getting traces out of your tools

| Source | How to export |
|---|---|
| **Jaeger UI** | Open the trace → *Download JSON* (`{"data": [...]}`). Paste it as is inside `[ … ]`. |
| **Jaeger API** | `curl "$JAEGER/api/traces/<traceID>"`, or on Jaeger v2 `curl "$JAEGER/api/v3/traces/<traceID>"` (OTLP inside `{"result": …}`). |
| **Grafana Tempo** | `curl "$TEMPO/api/traces/<traceID>"` (`{"batches": [...]}`) or `curl "$TEMPO/api/v2/traces/<traceID>"` (`{"trace": …}`). Both work as is. |
| **Zipkin** | `curl "$ZIPKIN/api/v2/trace/<traceID>"` (a span array) or `/api/v2/traces?...` (array of arrays). |
| **OpenTelemetry Collector** | The `file` exporter (`format: json`) writes one `ExportTraceServiceRequest` per line. `jq -s '{traces: .}' traces.json` turns the file into an input; a trace split across several lines is joined back together by trace ID. |
| **OTLP/HTTP JSON** | Any `POST /v1/traces` JSON body you captured. |
| **Anything else** | Map to the simplified shape (below). |

#### Via the API (curl)

```bash
curl "$JAEGER/api/traces/4bf92f3577b34da6a3ce929d0e0e4736" > trace.json
jq -n --slurpfile t trace.json '{traces: $t, latencySloMs: 1000}' |
  curl -s -X POST -H "Content-Type: application/json" -d @- \
  "https://api.apify.com/v2/acts/michiganpublictech~trace-root-cause-analyzer/run-sync-get-dataset-items?token=$APIFY_TOKEN" |
  jq '.[0].data.suspected_root_causes[] | {rank, service, operation, category, confidence, reasoning}'
```

#### Python (apify-client)

```python
import json
from pathlib import Path
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("michiganpublictech/trace-root-cause-analyzer").call(
    run_input={
        "traces": [json.loads(Path("incident-trace.json").read_text())],
        "baselineTraces": [json.loads(Path("healthy-traces.json").read_text())],
        "latencySloMs": 800,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    d = item["data"]
    if item["status"] != "ok" or d.get("kind") == "fleet-summary":
        continue  # a failed (uncharged) item or the fleet summary
    print(item["target"], d["trace_duration_ms"], "ms", "FAILED" if d["request_failed"] else "")
    for rc in d["suspected_root_causes"]:
        print(f"  #{rc['rank']} {rc['service']} › {rc['operation']} [{rc['category']}, {rc['confidence']}]")
        print("     ", rc["reasoning"][0])
        print("      next:", rc["recommendations"][0])
print(client.key_value_store(run["defaultKeyValueStoreId"]).get_record("INVESTIGATION.md")["value"])
```

#### In CI or synthetic checks

Set `failOnSeverity: "high"` to make the run fail when a trace has a high-severity finding, for example a load-test trace with a new N+1 query or a regression against the baseline. `REPORT.sarif` integrates with code-scanning dashboards.

### Input

#### Supported trace formats (auto-detected, can be mixed)

**OTLP/JSON** (`ExportTraceServiceRequest`, the OpenTelemetry wire format). IDs can be hex (as the spec requires) or base64 (canonical protobuf JSON, as Tempo returns them). Timestamps and integer attributes can be strings or numbers. `kind` and `status.code` can be integers or enum names (`SPAN_KIND_SERVER`, `STATUS_CODE_ERROR`). Array and key-value-list attribute values are kept. Also accepted: Tempo's `{"batches": [...]}` and `{"trace": {...}}`, Jaeger v2's `{"result": {...}}`, the legacy `instrumentationLibrarySpans`, and JSON-lines text (one request per line, as the Collector `file` exporter writes).

```json
{"resourceSpans": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "orders"}}]},
  "scopeSpans": [{"spans": [{"traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "00f067aa0ba902b7",
    "parentSpanId": "53995c3f42cd8ad8", "name": "SELECT shop.orders", "kind": 3,
    "startTimeUnixNano": "1760000000016000000", "endTimeUnixNano": "1760000001316000000",
    "attributes": [{"key": "db.system.name", "value": {"stringValue": "postgresql"}}],
    "status": {"code": 0}}]}]}]}
```

**Jaeger JSON** (UI download / query API): `{"data": [{"traceID", "spans": [...], "processes": {...}}]}`, or a single trace object. Times are in µs. A Jaeger error response (`"errors": [{"msg": "trace not found"}]`) is reported as a failed, uncharged item with that message. `CHILD_OF` and `FOLLOWS_FROM` references are understood: `FOLLOWS_FROM` children count as asynchronous, and extra references become links. Tags (`error`, `span.kind`, `otel.status_code`) and logs (exception events) are mapped. Short IDs with leading zeros stripped are padded back.

**Zipkin v2 JSON**: a span array (or an array of traces). Times are in µs. `remoteEndpoint.serviceName` becomes `peer.service`. Shared B3 single-host server spans that reuse the client's span ID are split into their own span.

**Simplified** for anything else:

```json
{"trace_id": "checkout-42", "spans": [
  {"span_id": "a1", "service": "frontend", "name": "GET /api/checkout", "kind": "server", "start_ms": 0, "duration_ms": 2350},
  {"span_id": "d1", "parent_span_id": "a1", "service": "orders", "name": "SELECT shop.inventory", "kind": "client",
   "start_ms": 295, "duration_ms": 1450, "status": "error", "status_message": "statement timeout",
   "attributes": {"db.system.name": "postgresql", "db.query.text": "SELECT * FROM inventory WHERE lower(sku) = $1"}}
]}
```

CamelCase keys (`spanId`, `parentSpanId`, `startMs`, `durationMs`), `end_ms` instead of `duration_ms`, `"status": {"code": "ERROR", "message": "..."}`, `events`, `links` and `"follows_from": true` are also accepted.

**Durations-only mode.** If spans have no `start_ms`, the trace is still analysed, but timing is approximate. Children are assumed sequential when their durations fit inside the parent; otherwise the largest are taken greedily (the "heaviest path"). Timing-dependent detectors (gaps, network/queue time, retry backoff, consumer lag, clock skew, sequential-call detection) are skipped, the critical path is flagged `critical_path_approximate: true`, and suspect scores are multiplied by 0.75.

A single payload can contain many trace IDs. Spans are grouped by trace ID and each trace is analysed (and billed) separately.

#### Input reference

| Field | Type | Default | Description |
|---|---|---|---|
| `traces` | array (JSON) | demo trace (prefill) | **Required.** Trace payloads in any supported format. A bare Zipkin span array can be pasted as is. |
| `baselineTraces` | array (JSON) | — | Healthy traces of the same request type, used for regression analysis. |
| `latencySloMs` | integer | — | Latency objective. Adds SLO-breach findings and the fleet breach rate. |
| `focusService` | string | — | Restricts the analysis window to this service's top-most span and everything it calls. Traces without the service are skipped and not charged. |
| `topRootCauses` | integer 1–10 | `5` | Number of ranked suspects per trace. |
| `maxTraces` | integer 1–1000 | `25` | Maximum number of traces to analyse, in input order. |
| `thresholds` | object | — | Per-run overrides for detector thresholds, e.g. `{"slowDbMs": 250}`. See [Tuning thresholds](#tuning-thresholds). |
| `includeWaterfall` | boolean | `true` | Include the ASCII waterfall. |
| `fleetSummary` | boolean | `true` | Add the aggregate item when 2 or more traces are investigated. |
| `failOnSeverity` | `none`/`critical`/`high`/`medium`/`low` | `none` | CI gate: fail the run at this finding severity. |
| `disabledRules` | array of strings | `[]` | Rule IDs to skip (see [RULES.md](RULES.md)). Requires a plan with custom rules. |
| `licenseKey` | secret string | — | Optional enterprise or prepaid license key. |

### How it works

The pipeline is made of pure, separately tested modules: **parse/normalize → tree → timing → critical path → errors and patterns → baseline → detectors → ranking → report**.

1. **Normalize.** Every format is converted to one span model with integer **nanosecond** timestamps, a service name, kind, status, attributes, events and links. Current and legacy OpenTelemetry semantic conventions are both read (`http.response.status_code`/`http.status_code`, `db.query.text`/`db.statement`, `db.system.name`/`db.system` and so on). For grouping and baseline matching, span names are also reduced to a **template**: UUIDs, long hex IDs, opaque tokens, stand-alone numbers, e-mail addresses and query values become `{id}`, so `GET /users/8812` and `GET /users/8813` count as the same operation. Version-like parts (`/v1/`, `HTTP/1.1`) are kept. Outputs still show the concrete span name whenever the grouped spans share one.
2. **Rebuild the tree.** Parent/child links are rebuilt from the IDs, with these repairs:
   - Identical duplicate spans are dropped. Conflicting duplicates are renamed `id~dupN` and kept.
   - Cycles in parent references are broken.
   - Orphans (parent missing) become separate sub-trees.
   - The primary root is the longest real root. If there is none, it is the longest orphan sub-tree, and the trace is flagged incomplete.
   - **Clock skew** is corrected Jaeger-style. A child from a *different* service that starts before or ends after its parent is shifted, together with its sub-tree, so that it sits centred in the parent. Same-service children are never shifted because they share a clock. Asynchronous children (FOLLOWS_FROM, producer→consumer) may outlive their parent and are only corrected if they start before it.
   - CLIENT→SERVER pairs across services are matched for network timing.
3. **Timing.** A span's **self (exclusive) time** is its duration minus the *interval union* of its synchronous children, clipped to the span. With a union instead of a sum, two parallel 100 ms calls inside a 120 ms parent leave 20 ms of self time, not −80 ms. Other measures:
   - **Gaps**: intervals of a parent that no child covers.
   - **Network/queue time**: client duration minus server duration, split into before and after the server ran.
   - **Parallelism factor**: total self time ÷ wall time.
   - Aggregates per service, per operation and per dependency edge.
4. **Critical path.** A backwards walk from the end of the root. At each step it picks the synchronous child that finished last before the cursor, assigns the gap after that child to the parent, recurses into the child, then moves the cursor to the child's start. Concurrent children hidden behind a longer sibling are never selected. Overflowing children are clipped. The walk is iterative, so very deep traces are safe. Every nanosecond of the root window is attributed to exactly one span, so per-span and per-service shares add up to 100 %.
5. **Error origin vs. propagation.** Errors are status ERROR, the `error` tag, or HTTP 5xx. The walk starts at the failed request and, at each level, follows the **last-finishing** failing child, because earlier failures were usually retried. It passes through intermediate spans that forgot to set an error status. The end of that chain is the **origin**. Errors above it are *propagated*, and errors whose parent did not fail are *handled*. Exception events (type and message) are attached, with secrets redacted.
6. **Detectors.** 56 rules turn patterns into findings. Each finding names the culprit span, gives a severity, evidence and a recommendation. See [RULES.md](RULES.md). Sibling calls are grouped by service, span-name template, remote target and query shape, which is how N+1 loops and retries are recognized even when every call has a different ID in its name.
7. **Ranking.** Candidates are `(service, operation)` pairs, scored on four signals between 0 and 1:

   - **impact** = max(critical-path share ÷ 0.5, error-origin credit). The credit is 1.0 for the origin of the request's failure. When the origin is a *timeout* on an outgoing call, 80 % of its credit goes to the slowest work under that call, because the timeout is a symptom.
   - **evidence** = noisy-OR of the strongest finding per rule. The weights are high 0.6, medium 0.35 and low 0.15, each multiplied by the finding's confidence.
   - **baseline deviation** = log2(slowdown) ÷ 3 (an 8× slowdown gives 1.0), 0.7 for new errors and 0.5 for new operations.
   - **score** = (0.6·impact + 0.25·evidence + 0.15·deviation), normalized when there is no baseline, then multiplied by a data-quality factor: durations-only 0.75, missing root 0.85, many orphans 0.9. Confidence is **high** at 0.6 or above, **medium** at 0.35 or above, and **low** otherwise.

   The **category** of a suspect comes from its most specific detectors, such as `RCA-DB-001` → database or `RCA-RES-001` → resource. Location-only detectors like "dominant span" are used only when nothing specific fired. Each suspect's recommendations combine the detectors' recommendations with category-specific next steps.
8. **Baseline comparison** (optional). Operations are matched on service and span-name template (see step 1), using robust statistics. A **regression** needs all three of the following: an incident median at least 2× the baseline median, above 1.2× the baseline p95, and at least 20 ms slower. Also reported: critical-path share shifts of 25 percentage points or more, new operations and services, new errors, call-count increases of 2× or more, and the end-to-end slowdown. A trace is compared (and charged) only if the baseline contains its root operation or at least 20 % of its operations.

#### Root-cause categories

| Category | Typical evidence | Typical next step |
|---|---|---|
| `database` | Slow query, DB error, deadlock or lock timeout | Query plan (EXPLAIN ANALYZE), indexes, locks, DB saturation |
| `code` | N+1, sequential awaits, CPU-bound span, uninstrumented gap, exception origin, cache stampede, chatty calls | Batch or eager-load, parallelize, profile, fix the bug |
| `dependency` | 502/503/504 or UNAVAILABLE from an internal service, slow cache | The callee's own traces, health and deploys; circuit breakers |
| `network` | Client ≫ server time, slow DNS/TCP/TLS, connection refused or reset | Proxies and meshes, connection reuse, DNS, network policies |
| `resource` | Pool wait or exhaustion, GC, cold start, OOM, throttling (429), consumer lag | Pool and limit sizing, provisioned concurrency, scaling |
| `config` | Retries without backoff, retry storms, timeouts at round values, auth failures, cache TTLs | Timeout and retry budgets, backoff with jitter, credentials |
| `external` | Slow or failing third-party host (not one of your services) | Provider status, timeouts and circuit breaker, caching, async |

### Output

One dataset item per trace (a standard Report: `score`, `risk_level`, `summary`, `findings`, plus everything below in `data`), and one optional **fleet summary** item. Abridged example for the retry-storm demo:

```json
{
  "product": "trace-root-cause-analyzer",
  "target": "trace c5c0bd655b7ebfbf636581530c19aaa7",
  "status": "ok",
  "score": 61,
  "risk_level": "medium",
  "summary": {"critical": 0, "high": 11, "medium": 5, "low": 0, "info": 0},
  "findings": [{"rule_id": "RCA-ERR-001", "severity": "high", "kind": "observation",
                "location": "payment › POST /charge [000004ccdd70e019]",
                "description": "The failure originates in payment › POST /charge and propagated through 4 ancestor span(s) to the request.",
                "evidence": {"service": "payment", "operation": "POST /charge",
                             "error": {"message": "service unavailable: payment-db failover in progress", "http_status": 503}}}],
  "data": {
    "trace_id": "c5c0bd655b7ebfbf636581530c19aaa7",
    "input_formats": ["jaeger-json"],
    "root_service": "gateway", "root_operation": "POST /api/checkout",
    "trace_duration_ms": 620.0, "span_count": 25, "services": ["gateway", "checkout", "payment"],
    "request_failed": true, "timing_mode": "timestamps", "parallelism": 1.0,
    "slo": {"slo_ms": 500, "breached": true, "over_by_ms": 120.0},
    "suspected_bottleneck": {"service": "payment", "operation": "POST /charge", "self_ms": 44.0, "share": 0.071},
    "suspected_root_causes": [
      {"rank": 1, "service": "payment", "operation": "POST /charge", "category": "dependency",
       "confidence": "high", "score": 0.977, "critical_path_ms": 396.0, "critical_path_share": 0.639,
       "error_origin": true,
       "reasoning": [
         "Holds 396 ms of exclusive time on the critical path (64% of 620 ms).",
         "Origin of the error that failed the request.",
         "RCA-ERR-002 (high): An error in payment propagated through 3 services: payment → checkout → gateway.",
         "RCA-HTTP-002 (high): payment returned HTTP 503 9 time(s) (on the error chain)."],
       "recommendations": ["The callee (or the proxy in front of it) was unavailable or overloaded: check its health, ..."],
       "signals": {"impact": 1.0, "critical_path_share": 0.639, "error_origin": 1.0, "evidence": 0.922,
                   "baseline_deviation": 0.0, "data_quality_factor": 1.0}},
      {"rank": 2, "service": "checkout", "operation": "POST /charge", "category": "config", "confidence": "medium",
       "reasoning": ["...", "RCA-RETRY-002 (high): 8 retries across 2 retrying call site(s); worst amplification ×9 ..."]}
    ],
    "critical_path": [{"service": "gateway", "operation": "POST /api/checkout", "span_id": "000004ccdd70e001",
                       "duration_ms": 620.0, "self_ms": 20.0, "share": 0.0323}],
    "critical_path_by_service": [{"service": "payment", "ms": 396.0, "share": 0.639}],
    "error_chain": [{"role": "request", "service": "gateway", "operation": "POST /api/checkout", "http_status": 503},
                    {"role": "origin", "service": "payment", "operation": "POST /charge",
                     "message": "service unavailable: payment-db failover in progress", "http_status": 503}],
    "dependency_graph": {"nodes": [...], "edges": [{"source": "checkout", "target": "payment", "calls": 9, "errors": 9, "p50_ms": 52.0}]},
    "baseline": null,
    "recommendations": ["..."],
    "waterfall": "span   duration  |0 ... 620 ms|\ngateway › POST /api/checkout  620 ms |#====...",
    "critical_path_mermaid": "flowchart LR\n  n0[\"gateway<br/>POST /api/checkout<br/>20 ms · 3%\"] ...",
    "data_quality": {"timed": true, "orphan_spans": 0, "clock_skew_adjusted_spans": 0, "caveats": []}
  },
  "disclaimer": "Heuristic analysis of the spans you supplied. ..."
}
```

Field notes:

- **`suspected_root_causes`**: ranked suspects. Each has `category`, `confidence` (`high`/`medium`/`low`), a numeric `score` (0–1), the raw `signals`, the `reasoning` (why it was ranked), the detector `evidence`, the `span_ids` involved and `recommendations`.
- **`critical_path`**: spans on the path in time order. `self_ms` is the time attributed to that span on the path, and `share` is its fraction of `trace_duration_ms`. Shares add up to 1.
- **`error_chain`**: from the failed request (`request`) through `propagated` spans to the `origin`, with messages, status codes and exception type/message (secrets redacted).
- **`dependency_graph`**: services, databases, caches, queues and external hosts, with call counts, errors and latency per edge.
- **`baseline`**: present when a comparison was made. Includes `regressions`, `critical_path_shifts`, `new_operations`, `new_services`, `new_errors`, `call_count_increases` and the end-to-end comparison.
- **`data_quality`**: orphans, roots, skew adjustments, duplicates, cycles, plus human-readable `caveats` (also listed in `errors` so they appear in REPORT.md).
- **`waterfall`**: a fixed-width timeline. `#` means on the critical path, `=` the rest of the span, `!` an error. Large traces are trimmed to the root, error chain, critical path and slowest spans.
- **Fleet summary item** (`data.kind = "fleet-summary"`): `top_root_causes`, `top_bottlenecks`, `top_error_origins`, `operation_latency` (p50/p95/p99 per operation across traces), `duration_ms` percentiles, SLO breach rate, and `RCA-FLEET-*` findings.
- **Failed items** (`status: "failed"`) explain why a payload or trace could not be analysed. **They are never charged.**
- **`score`/`risk_level`** follow the shared scoring of all our Actors (100 = no findings). For traces they are a rough "how many problems" signal; read the suspects for the diagnosis.

**`INVESTIGATION.md`** (key-value store) contains the same analysis as a readable report: the fleet table, then per trace the suspects with reasoning and next steps, critical-path table, Mermaid critical path, error chain, waterfall, Mermaid dependency graph and caveats. It renders well on GitHub, GitLab and in most incident tools.

### Pricing

Pay per event. You only pay for traces that were actually analysed.

| Event | Price | When |
|---|---|---|
| `trace-investigated` | **$0.50** | Per trace analysed (primary event). |
| `baseline-comparison` | **$0.25** | Per trace that was compared with your `baselineTraces`. Only charged when the baseline matches the trace (its root operation, or at least 20 % of its operations). |
| `fleet-summary` | **$0.25** | At most once per run, for the aggregate item. See below. |

**The fleet summary is a separate, paid item.** When a run investigates (and charges) 2 or more traces, the Actor adds one extra dataset item that aggregates them: recurring root causes and bottlenecks, latency percentiles per operation, error origins, the SLO breach rate and the `RCA-FLEET-*` findings. It is charged once, $0.25, no matter how many traces it covers. It is on by default; set `fleetSummary: false` to skip it and its charge. The per-trace results are the same either way. Runs with a single analysed trace never produce or charge it.

Examples:

| Run | Cost |
|---|---|
| 1 trace | 1 × $0.50 = **$0.50** |
| 3 traces, no baseline | 3 × $0.50 + $0.25 fleet summary = **$1.75** (or $1.50 with `fleetSummary: false`) |
| 10 traces with a matching baseline | 10 × $0.50 + 10 × $0.25 + $0.25 = **$7.75** |

**Never charged:** unparseable payloads, traces with no usable spans or a zero-duration root, traces skipped by `focusService`, and traces beyond `maxTraces`. If your spending limit is reached, the Actor stops *before* starting work it can't bill: it skips the baseline comparison or the fleet summary first, then stops.

### Tuning thresholds

Every number a detector compares against has a default, and you can override any of them for one run with the `thresholds` input object. For example, a batch job where 300 ms queries are normal and only loops of 10 or more calls are worth flagging:

```json
{"thresholds": {"slowDbMs": 1000, "verySlowDbMs": 5000, "nPlusOneMin": 10}}
```

Unknown keys and out-of-range values fail the run with an error before anything is charged. `ms` values are milliseconds; ratios are between 0 and 1; counts are whole numbers. The overrides used are listed in the run's `OUTPUT` notes.

| Key | Default | Rules | What it sets |
|---|---|---|---|
| `dominantSpanShare` | 0.5 | RCA-LAT-001 | Share of the critical path one span must hold. |
| `dominantSpanHighShare` | 0.7 | RCA-LAT-001 | Share at which it becomes high severity. |
| `dominantSpanMinMs` | 50 ms | RCA-LAT-001/002 | Minimum critical-path time of the dominant span or service. |
| `dominantServiceShare` | 0.75 | RCA-LAT-002 | Share of the critical path one service must hold. |
| `gapMinMs` | 50 ms | RCA-LAT-004 | Minimum uninstrumented gap. |
| `gapMinFraction` | 0.2 | RCA-LAT-004 | Minimum gap as a fraction of the parent span. |
| `cpuLeafMinMs` | 100 ms | RCA-LAT-005 | Minimum self time of a CPU-bound leaf span. |
| `cpuLeafShare` | 0.25 | RCA-LAT-005 | Minimum critical-path share of that leaf. |
| `nPlusOneMin` | 5 | RCA-CODE-001, RCA-DB-002 | Repeated calls/queries that count as N+1. |
| `nPlusOneHigh` | 20 | RCA-CODE-001, RCA-DB-002 | Repeats that make N+1 high severity. |
| `sequentialSavingMinMs` | 50 ms | RCA-CODE-002 | Minimum time saved by parallelizing sequential calls. |
| `sequentialSavingFraction` | 0.15 | RCA-CODE-002 | Minimum saving as a fraction of the trace. |
| `fanOutChildren` | 50 | RCA-CODE-003 | Children of one span that count as a fan-out. |
| `fanOutConcurrency` | 25 | RCA-CODE-003 | Peak concurrent children that count as a fan-out. |
| `fanOutHigh` | 200 | RCA-CODE-003 | Children that make a fan-out high severity. |
| `slowDbMs` | 100 ms | RCA-DB-001 | Duration of a slow database call. |
| `verySlowDbMs` | 500 ms | RCA-DB-001 | Duration that makes it high severity. |
| `poolWaitMs` | 20 ms | RCA-DB-004, RCA-RES-001 | Connection-pool wait worth reporting. |
| `networkOverheadMs` | 50 ms | RCA-NET-001 | Client time not spent in the server. |
| `networkOverheadFraction` | 0.3 | RCA-NET-001 | That overhead as a fraction of the client span. |
| `connectSlowMs` | 50 ms | RCA-NET-002 | Slow DNS/TCP/TLS connection setup. |
| `externalSlowMs` | 200 ms | RCA-EXT-001 | Slow third-party call. |
| `retryStormAmplification` | 4 | RCA-RETRY-002 | Multiplied attempts across nested retry layers. |
| `retryStormTotal` | 6 | RCA-RETRY-002 | Retries in one request. |
| `noBackoffGapMs` | 10 ms | RCA-RETRY-003 | Largest gap between attempts that still means no backoff. |
| `timeoutClusterMinMs` | 250 ms | RCA-TIMEOUT-002 | Minimum duration of spans checked for round-number timeouts. |
| `cacheMissRatio` | 0.5 | RCA-CACHE-001 | Cache miss ratio. |
| `cacheMinLookups` | 3 | RCA-CACHE-001 | Lookups needed before the miss ratio is judged. |
| `slowCacheMs` | 50 ms | RCA-CACHE-003 | Slow cache call. |
| `consumerLagMs` | 500 ms | RCA-MSG-001 | Producer-to-consumer lag. |
| `consumerLagHighMs` | 5,000 ms | RCA-MSG-001 | Lag that makes it high severity. |
| `chattyCalls` | 20 | RCA-TOPO-001 | Calls between two services that count as chatty. |
| `largeTraceSpans` | 1,000 | RCA-TOPO-002 | Spans that make a trace unusually large. |

### Rules

56 per-trace rules and 4 fleet rules, with severities, descriptions and recommendations, are listed in **[RULES.md](RULES.md)**. Families: `LAT` latency, `CODE` call patterns, `DB` databases, `RES` resources, `NET` network, `EXT` external dependencies, `HTTP` status codes, `RETRY`, `TIMEOUT`, `ERR` error origin, `CACHE`, `MSG` messaging, `TOPO` topology, `DATA` data quality, `BASE` baseline, `ROOT` top suspect, `FLEET` aggregate. Rule IDs are stable. Use them in `disabledRules`.

### FAQ

**Does it send my traces to an AI model or anywhere else?**
No. The analysis is deterministic code running inside the Actor, and nothing is sent to other services. Secrets found in error messages and query text are redacted with the same detector our security Actors use. Literal values in SQL are replaced by `?`.

**How is this different from the waterfall in Jaeger, Tempo or Datadog?**
Those show the data. This interprets it: it computes the critical path with concurrency handled correctly, separates error origin from propagation, runs 56 failure-pattern detectors, compares with healthy traces and ranks causes with explicit reasoning. You can use the output as a first-pass triage and verify it in your trace UI.

**What if my spans have no start times?**
You get *durations-only mode*: a self-time / heaviest-path approximation with lower confidence (see "Durations-only mode" above). For the full analysis, export spans with timestamps.

**My services' clocks are not in sync.**
Cross-service skew is detected and corrected Jaeger-style (`RCA-DATA-003`, `data_quality.clock_skew_adjusted_spans`). The network/queue split is then omitted for the adjusted calls, because it would be unreliable.

**The trace is partial (missing root or dropped spans).**
The longest available sub-tree is analysed. Orphans and missing roots are reported, and confidence is reduced.

**Can I analyse many traces at once?**
Yes. Pass several payloads, or one payload with many trace IDs. Each trace is a separate item, and with 2 or more you also get the fleet summary. Use `maxTraces` to cap the cost.

**Which semantic conventions are understood?**
Current and legacy OpenTelemetry HTTP, RPC/gRPC, database, messaging, FaaS and exception conventions, plus common attributes such as `peer.service`, `server.address`, `cache.hit`, `http.request.resend_count`, `db.client.connection.pending_requests` and `faas.coldstart`. Jaeger and Zipkin tags (`error`, `span.kind`, `http.status_code`) are mapped too.

### Limitations and responsible use

- **Hypotheses, not proof.** A trace only shows what was instrumented and sampled. A slow span can be the victim of a noisy neighbour, a saturated node or a lock held by another request that is not in the trace. Confirm with metrics and logs.
- **Missing instrumentation** shows up as self time or gaps. The Actor reports it (`RCA-LAT-004`, `RCA-DATA-006`) but cannot see inside it.
- **Detectors are heuristics.** Their default thresholds (listed under [Tuning thresholds](#tuning-thresholds) and in RULES.md) were chosen by hand, not fitted to your traffic. Some findings may not matter for your system: raise the threshold with `thresholds` or turn the rule off with `disabledRules`. The ranking weights and confidence cut-offs are fixed.
- **Durations-only and partial traces** produce lower-confidence results, labelled as such.
- **Limits:** up to 50,000 spans per trace and `maxTraces` traces per run (plus your plan's per-run target limit). Very large inputs need more memory; the default is 1 GB.
- Only submit traces you are allowed to process. Avoid sending personal data in span attributes. The Actor redacts credentials it recognizes, but it cannot know every kind of sensitive value.

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

# Changelog

This Actor's version history is a separate document: https://apify.com/michiganpublictech/trace-root-cause-analyzer/changelog.md

# Actor input Schema

## `traces` (type: `array`):

A list of trace payloads in any supported format: OTLP/JSON ExportTraceServiceRequest ({"resourceSpans": \[...]}, hex or base64 ids; Tempo {"batches"}/{"trace"} and Jaeger v2 {"result"} responses too), Jaeger JSON ({"data": \[...]}), a Zipkin v2 span array, or the simplified shape {"trace_id", "spans": \[{"span_id", "parent_span_id", "service", "name", "start_ms", "duration_ms", "status", "kind", "attributes"}]}. A bare Zipkin span array can be pasted as-is; OTel Collector JSON-lines output can be given as one string. Without start_ms the analysis runs in a lower-confidence durations-only mode.

## `baselineTraces` (type: `array`):

Optional healthy traces of the same request type, in any supported format. Enables per-operation latency regression (robust median/p95), critical-path share shifts, new operations/services/errors and call-count increases. Operations are matched by service and span-name template (IDs in span names are ignored). Charged as 'baseline-comparison' per trace only when the baseline matches the trace's operations.

## `latencySloMs` (type: `integer`):

Optional latency objective for the request. Traces slower than this get an SLO-breach finding, and the fleet summary reports the breach rate.

## `focusService` (type: `string`):

Optional service name. The analysis window becomes this service's top-most span (the service and everything it calls). Traces without this service are skipped and not charged.

## `topRootCauses` (type: `integer`):

How many ranked suspects to return per trace.

## `maxTraces` (type: `integer`):

Analyse at most this many traces (in input order). Protects your budget when an export contains many traces.

## `thresholds` (type: `object`):

Optional per-run overrides for detector thresholds, e.g. {"slowDbMs": 250, "nPlusOneMin": 10}. Unknown keys or out-of-range values fail the run before anything is charged. Ratios are 0-1. Keys and defaults: dominantSpanShare (0.5); dominantSpanHighShare (0.7); dominantSpanMinMs (50 ms); dominantServiceShare (0.75); gapMinMs (50 ms); gapMinFraction (0.2); cpuLeafMinMs (100 ms); cpuLeafShare (0.25); nPlusOneMin (5); nPlusOneHigh (20); sequentialSavingMinMs (50 ms); sequentialSavingFraction (0.15); fanOutChildren (50); fanOutConcurrency (25); fanOutHigh (200); slowDbMs (100 ms); verySlowDbMs (500 ms); poolWaitMs (20 ms); networkOverheadMs (50 ms); networkOverheadFraction (0.3); connectSlowMs (50 ms); externalSlowMs (200 ms); retryStormAmplification (4); retryStormTotal (6); noBackoffGapMs (10 ms); timeoutClusterMinMs (250 ms); cacheMissRatio (0.5); cacheMinLookups (3); slowCacheMs (50 ms); consumerLagMs (500 ms); consumerLagHighMs (5000 ms); chattyCalls (20); largeTraceSpans (1000). See README "Tuning thresholds" for what each one controls.

## `includeWaterfall` (type: `boolean`):

Add a compact text waterfall (critical path highlighted) to each result and to INVESTIGATION.md.

## `fleetSummary` (type: `boolean`):

When two or more traces are investigated, add one aggregate item: recurring root causes and bottlenecks, latency percentiles per operation, error origins and SLO breach rate. Charged once per run as 'fleet-summary' ($0.25). Untick to skip it and its charge; per-trace results are unaffected.

## `failOnSeverity` (type: `string`):

Fail the run (non-zero exit, useful in CI or synthetic checks) when any finding is at or above this severity.

## `disabledRules` (type: `array`):

Rule IDs to skip, e.g. RCA-CODE-002 (see RULES.md). Requires a plan with the custom rules feature.

## `licenseKey` (type: `string`):

Optional enterprise or prepaid license key. Leave empty to pay per event on Apify.

## Actor input object example

```json
{
  "traces": [
    {
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "spans": [
        {
          "span_id": "a1",
          "service": "frontend",
          "name": "GET /api/checkout",
          "kind": "server",
          "start_ms": 0,
          "duration_ms": 2350,
          "attributes": {
            "http.request.method": "GET",
            "http.route": "/api/checkout",
            "http.response.status_code": 200
          }
        },
        {
          "span_id": "b1",
          "parent_span_id": "a1",
          "service": "frontend",
          "name": "POST /orders",
          "kind": "client",
          "start_ms": 12,
          "duration_ms": 2323,
          "attributes": {
            "peer.service": "orders",
            "http.response.status_code": 200
          }
        },
        {
          "span_id": "c1",
          "parent_span_id": "b1",
          "service": "orders",
          "name": "POST /orders",
          "kind": "server",
          "start_ms": 20,
          "duration_ms": 2300
        },
        {
          "span_id": "c2",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.customers",
          "kind": "client",
          "start_ms": 25,
          "duration_ms": 12,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM customers WHERE id = 42"
          }
        },
        {
          "span_id": "i0",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 40,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9001"
          }
        },
        {
          "span_id": "i1",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 82,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9002"
          }
        },
        {
          "span_id": "i2",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 124,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9003"
          }
        },
        {
          "span_id": "i3",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 166,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9004"
          }
        },
        {
          "span_id": "i4",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 208,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9005"
          }
        },
        {
          "span_id": "i5",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 250,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9006"
          }
        },
        {
          "span_id": "d1",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.inventory",
          "kind": "client",
          "start_ms": 295,
          "duration_ms": 1450,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM inventory WHERE lower(sku) = 'sku-123'"
          }
        },
        {
          "span_id": "e1",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "POST /charge",
          "kind": "client",
          "start_ms": 1750,
          "duration_ms": 550,
          "attributes": {
            "peer.service": "payment",
            "http.response.status_code": 200
          }
        },
        {
          "span_id": "e2",
          "parent_span_id": "e1",
          "service": "payment",
          "name": "POST /charge",
          "kind": "server",
          "start_ms": 1760,
          "duration_ms": 530
        },
        {
          "span_id": "e3",
          "parent_span_id": "e2",
          "service": "payment",
          "name": "POST /v1/charges",
          "kind": "client",
          "start_ms": 1770,
          "duration_ms": 500,
          "attributes": {
            "server.address": "api.stripe.com",
            "http.response.status_code": 200
          }
        }
      ]
    }
  ],
  "latencySloMs": 1000,
  "topRootCauses": 5,
  "maxTraces": 25,
  "includeWaterfall": true,
  "fleetSummary": true,
  "failOnSeverity": "none"
}
```

# Actor output Schema

## `results` (type: `string`):

One item per trace: suspected root causes, critical path, error chain, dependency graph, findings and data quality.

## `investigation` (type: `string`):

Human-readable root-cause write-up with critical-path tables, Mermaid diagrams and waterfalls (INVESTIGATION.md).

## `summary` (type: `string`):

Plan, per-trace scores and billing summary for the run (OUTPUT record).

## `reportMarkdown` (type: `string`):

All findings as Markdown (REPORT.md).

## `reportSarif` (type: `string`):

SARIF 2.1.0 report for CI tools (REPORT.sarif, on plans that include SARIF).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "traces": [
        {
            "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
            "spans": [
                {
                    "span_id": "a1",
                    "service": "frontend",
                    "name": "GET /api/checkout",
                    "kind": "server",
                    "start_ms": 0,
                    "duration_ms": 2350,
                    "attributes": {
                        "http.request.method": "GET",
                        "http.route": "/api/checkout",
                        "http.response.status_code": 200
                    }
                },
                {
                    "span_id": "b1",
                    "parent_span_id": "a1",
                    "service": "frontend",
                    "name": "POST /orders",
                    "kind": "client",
                    "start_ms": 12,
                    "duration_ms": 2323,
                    "attributes": {
                        "peer.service": "orders",
                        "http.response.status_code": 200
                    }
                },
                {
                    "span_id": "c1",
                    "parent_span_id": "b1",
                    "service": "orders",
                    "name": "POST /orders",
                    "kind": "server",
                    "start_ms": 20,
                    "duration_ms": 2300
                },
                {
                    "span_id": "c2",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.customers",
                    "kind": "client",
                    "start_ms": 25,
                    "duration_ms": 12,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM customers WHERE id = 42"
                    }
                },
                {
                    "span_id": "i0",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 40,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9001"
                    }
                },
                {
                    "span_id": "i1",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 82,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9002"
                    }
                },
                {
                    "span_id": "i2",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 124,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9003"
                    }
                },
                {
                    "span_id": "i3",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 166,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9004"
                    }
                },
                {
                    "span_id": "i4",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 208,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9005"
                    }
                },
                {
                    "span_id": "i5",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 250,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9006"
                    }
                },
                {
                    "span_id": "d1",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.inventory",
                    "kind": "client",
                    "start_ms": 295,
                    "duration_ms": 1450,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM inventory WHERE lower(sku) = 'sku-123'"
                    }
                },
                {
                    "span_id": "e1",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "POST /charge",
                    "kind": "client",
                    "start_ms": 1750,
                    "duration_ms": 550,
                    "attributes": {
                        "peer.service": "payment",
                        "http.response.status_code": 200
                    }
                },
                {
                    "span_id": "e2",
                    "parent_span_id": "e1",
                    "service": "payment",
                    "name": "POST /charge",
                    "kind": "server",
                    "start_ms": 1760,
                    "duration_ms": 530
                },
                {
                    "span_id": "e3",
                    "parent_span_id": "e2",
                    "service": "payment",
                    "name": "POST /v1/charges",
                    "kind": "client",
                    "start_ms": 1770,
                    "duration_ms": 500,
                    "attributes": {
                        "server.address": "api.stripe.com",
                        "http.response.status_code": 200
                    }
                }
            ]
        }
    ],
    "latencySloMs": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("michiganpublictech/trace-root-cause-analyzer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "traces": [{
            "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
            "spans": [
                {
                    "span_id": "a1",
                    "service": "frontend",
                    "name": "GET /api/checkout",
                    "kind": "server",
                    "start_ms": 0,
                    "duration_ms": 2350,
                    "attributes": {
                        "http.request.method": "GET",
                        "http.route": "/api/checkout",
                        "http.response.status_code": 200,
                    },
                },
                {
                    "span_id": "b1",
                    "parent_span_id": "a1",
                    "service": "frontend",
                    "name": "POST /orders",
                    "kind": "client",
                    "start_ms": 12,
                    "duration_ms": 2323,
                    "attributes": {
                        "peer.service": "orders",
                        "http.response.status_code": 200,
                    },
                },
                {
                    "span_id": "c1",
                    "parent_span_id": "b1",
                    "service": "orders",
                    "name": "POST /orders",
                    "kind": "server",
                    "start_ms": 20,
                    "duration_ms": 2300,
                },
                {
                    "span_id": "c2",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.customers",
                    "kind": "client",
                    "start_ms": 25,
                    "duration_ms": 12,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM customers WHERE id = 42",
                    },
                },
                {
                    "span_id": "i0",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 40,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9001",
                    },
                },
                {
                    "span_id": "i1",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 82,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9002",
                    },
                },
                {
                    "span_id": "i2",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 124,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9003",
                    },
                },
                {
                    "span_id": "i3",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 166,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9004",
                    },
                },
                {
                    "span_id": "i4",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 208,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9005",
                    },
                },
                {
                    "span_id": "i5",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.order_items",
                    "kind": "client",
                    "start_ms": 250,
                    "duration_ms": 40,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM order_items WHERE order_id = 9006",
                    },
                },
                {
                    "span_id": "d1",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "SELECT shop.inventory",
                    "kind": "client",
                    "start_ms": 295,
                    "duration_ms": 1450,
                    "attributes": {
                        "db.system.name": "postgresql",
                        "db.query.text": "SELECT * FROM inventory WHERE lower(sku) = 'sku-123'",
                    },
                },
                {
                    "span_id": "e1",
                    "parent_span_id": "c1",
                    "service": "orders",
                    "name": "POST /charge",
                    "kind": "client",
                    "start_ms": 1750,
                    "duration_ms": 550,
                    "attributes": {
                        "peer.service": "payment",
                        "http.response.status_code": 200,
                    },
                },
                {
                    "span_id": "e2",
                    "parent_span_id": "e1",
                    "service": "payment",
                    "name": "POST /charge",
                    "kind": "server",
                    "start_ms": 1760,
                    "duration_ms": 530,
                },
                {
                    "span_id": "e3",
                    "parent_span_id": "e2",
                    "service": "payment",
                    "name": "POST /v1/charges",
                    "kind": "client",
                    "start_ms": 1770,
                    "duration_ms": 500,
                    "attributes": {
                        "server.address": "api.stripe.com",
                        "http.response.status_code": 200,
                    },
                },
            ],
        }],
    "latencySloMs": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("michiganpublictech/trace-root-cause-analyzer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "traces": [
    {
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "spans": [
        {
          "span_id": "a1",
          "service": "frontend",
          "name": "GET /api/checkout",
          "kind": "server",
          "start_ms": 0,
          "duration_ms": 2350,
          "attributes": {
            "http.request.method": "GET",
            "http.route": "/api/checkout",
            "http.response.status_code": 200
          }
        },
        {
          "span_id": "b1",
          "parent_span_id": "a1",
          "service": "frontend",
          "name": "POST /orders",
          "kind": "client",
          "start_ms": 12,
          "duration_ms": 2323,
          "attributes": {
            "peer.service": "orders",
            "http.response.status_code": 200
          }
        },
        {
          "span_id": "c1",
          "parent_span_id": "b1",
          "service": "orders",
          "name": "POST /orders",
          "kind": "server",
          "start_ms": 20,
          "duration_ms": 2300
        },
        {
          "span_id": "c2",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.customers",
          "kind": "client",
          "start_ms": 25,
          "duration_ms": 12,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM customers WHERE id = 42"
          }
        },
        {
          "span_id": "i0",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 40,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9001"
          }
        },
        {
          "span_id": "i1",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 82,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9002"
          }
        },
        {
          "span_id": "i2",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 124,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9003"
          }
        },
        {
          "span_id": "i3",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 166,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9004"
          }
        },
        {
          "span_id": "i4",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 208,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9005"
          }
        },
        {
          "span_id": "i5",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.order_items",
          "kind": "client",
          "start_ms": 250,
          "duration_ms": 40,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM order_items WHERE order_id = 9006"
          }
        },
        {
          "span_id": "d1",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "SELECT shop.inventory",
          "kind": "client",
          "start_ms": 295,
          "duration_ms": 1450,
          "attributes": {
            "db.system.name": "postgresql",
            "db.query.text": "SELECT * FROM inventory WHERE lower(sku) = '\''sku-123'\''"
          }
        },
        {
          "span_id": "e1",
          "parent_span_id": "c1",
          "service": "orders",
          "name": "POST /charge",
          "kind": "client",
          "start_ms": 1750,
          "duration_ms": 550,
          "attributes": {
            "peer.service": "payment",
            "http.response.status_code": 200
          }
        },
        {
          "span_id": "e2",
          "parent_span_id": "e1",
          "service": "payment",
          "name": "POST /charge",
          "kind": "server",
          "start_ms": 1760,
          "duration_ms": 530
        },
        {
          "span_id": "e3",
          "parent_span_id": "e2",
          "service": "payment",
          "name": "POST /v1/charges",
          "kind": "client",
          "start_ms": 1770,
          "duration_ms": 500,
          "attributes": {
            "server.address": "api.stripe.com",
            "http.response.status_code": 200
          }
        }
      ]
    }
  ],
  "latencySloMs": 1000
}' |
apify call michiganpublictech/trace-root-cause-analyzer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,michiganpublictech/trace-root-cause-analyzer"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/d72HKIprUVb84qGbO/builds/03ysONzwsB4hAN8dx/openapi.json