Distributed Trace Root Cause Analyzer

Paste a distributed trace (OpenTelemetry OTLP/JSON, Jaeger, Zipkin or a simple JSON shape) and get back why the request was slow or failed: the critical path, where the error originated, and a ranked list of suspected root causes, each with a confidence score, the evidence behind it and what to do next.

Trace UIs show you a waterfall of hundreds of spans. Working out from that waterfall which span actually made the request slow, which error caused the failure and which errors only propagated it, and whether you are looking at an N+1 loop, a retry storm, an exhausted connection pool or a cold start, takes an experienced engineer a while. This Actor does that triage automatically and deterministically, using the same reasoning an SRE would:

• Critical-path analysis that handles concurrent calls correctly, so you see the work that determined latency, not merely the longest spans.
• Error origin vs. propagation: it walks the error chain from the failed request down to the deepest failing span that explains the others.
• 56 trace detectors: slow and N+1 queries, sequential calls that could run in parallel, fan-out explosions, retries and retry storms, timeouts and timeout-budget inversions, connection-pool waits, lock contention, cache misses and stampedes, cold starts, GC hints, consumer lag, throttling, gateway errors, cascading failures, external dependency slowness, clock skew and more.
• Root-cause ranking that combines critical-path contribution, error origin, deviation from healthy baseline traces and detector evidence into an explained score (high / medium / low).
• Baseline comparison (optional): per-operation latency regressions with robust statistics (median/p95), critical-path shifts, and new operations, services and errors compared with healthy traces.
• Fleet summary across many traces (optional, one extra item per run): which service or operation is most often the root cause or bottleneck, latency percentiles per operation, and the SLO breach rate.

No LLM and no calls to your systems. The analysis is a pure function of the spans you provide, so the same input always gives the same answer.

⚠️ This is heuristic analysis of the spans you supplied, not proof. Missing instrumentation, sampling, clock skew and partial exports can hide the real cause. Every suspect comes with its reasoning and evidence so you can verify it with logs, metrics and the systems involved.

---

Who it's for

• On-call engineers and SREs: drop in the trace from the alert and get a first diagnosis in seconds, plus the next step to check.
• Platform and observability teams: run it on a batch of slow or failed traces from an incident to see the recurring cause, or put it in a pipeline after synthetic checks.
• Developers reviewing performance: catch N+1 queries, sequential awaits and chatty calls in traces from staging or load tests.
• AI agents and automations: the structured JSON output (suspected_root_causes, critical_path, error_chain) is designed to be consumed by other tools.

How to use

In Apify Console

1. Open the Actor and paste your trace into Traces (a list: wrap a single payload in [ … ]). The prefilled example is a checkout request with an N+1 query, a slow query and a slow third-party call.
2. Optionally add Baseline traces (healthy traces of the same request), a Latency SLO (ms) or a Focus service.
3. Click Start. Results:
   • Output → Overview / Root causes / Critical path / Findings tabs (one row per trace).
   • Key-value store: INVESTIGATION.md (the full write-up with critical-path tables, Mermaid diagrams, waterfalls and error chains), REPORT.md (findings), REPORT.sarif (on plans with SARIF) and OUTPUT (run summary).

Getting traces out of your tools

Source	How to export
Jaeger UI	Open the trace → Download JSON ({"data": [...]}). Paste it as is inside [ … ].
Jaeger API	curl "$JAEGER/api/traces/<traceID>", or on Jaeger v2 curl "$JAEGER/api/v3/traces/<traceID>" (OTLP inside {"result": …}).
Grafana Tempo	curl "$TEMPO/api/traces/<traceID>" ({"batches": [...]}) or curl "$TEMPO/api/v2/traces/<traceID>" ({"trace": …}). Both work as is.
Zipkin	curl "$ZIPKIN/api/v2/trace/<traceID>" (a span array) or /api/v2/traces?... (array of arrays).
OpenTelemetry Collector	The file exporter (format: json) writes one ExportTraceServiceRequest per line. jq -s '{traces: .}' traces.json turns the file into an input; a trace split across several lines is joined back together by trace ID.
OTLP/HTTP JSON	Any POST /v1/traces JSON body you captured.
Anything else	Map to the simplified shape (below).

Via the API (curl)

curl "$JAEGER/api/traces/4bf92f3577b34da6a3ce929d0e0e4736" > trace.json
jq -n --slurpfile t trace.json '{traces: $t, latencySloMs: 1000}' |
  curl -s -X POST -H "Content-Type: application/json" -d @- \
  "https://api.apify.com/v2/acts/michiganpublictech~trace-root-cause-analyzer/run-sync-get-dataset-items?token=$APIFY_TOKEN" |
  jq '.[0].data.suspected_root_causes[] | {rank, service, operation, category, confidence, reasoning}'

Python (apify-client)

import json
from pathlib import Path
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("michiganpublictech/trace-root-cause-analyzer").call(
    run_input={
        "traces": [json.loads(Path("incident-trace.json").read_text())],
        "baselineTraces": [json.loads(Path("healthy-traces.json").read_text())],
        "latencySloMs": 800,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    d = item["data"]
    if item["status"] != "ok" or d.get("kind") == "fleet-summary":
        continue  # a failed (uncharged) item or the fleet summary
    print(item["target"], d["trace_duration_ms"], "ms", "FAILED" if d["request_failed"] else "")
    for rc in d["suspected_root_causes"]:
        print(f"  #{rc['rank']} {rc['service']} › {rc['operation']} [{rc['category']}, {rc['confidence']}]")
        print("     ", rc["reasoning"][0])
        print("      next:", rc["recommendations"][0])
print(client.key_value_store(run["defaultKeyValueStoreId"]).get_record("INVESTIGATION.md")["value"])

In CI or synthetic checks

Set failOnSeverity: "high" to make the run fail when a trace has a high-severity finding, for example a load-test trace with a new N+1 query or a regression against the baseline. REPORT.sarif integrates with code-scanning dashboards.

Input

Supported trace formats (auto-detected, can be mixed)

OTLP/JSON (ExportTraceServiceRequest, the OpenTelemetry wire format). IDs can be hex (as the spec requires) or base64 (canonical protobuf JSON, as Tempo returns them). Timestamps and integer attributes can be strings or numbers. kind and status.code can be integers or enum names (SPAN_KIND_SERVER, STATUS_CODE_ERROR). Array and key-value-list attribute values are kept. Also accepted: Tempo's {"batches": [...]} and {"trace": {...}}, Jaeger v2's {"result": {...}}, the legacy instrumentationLibrarySpans, and JSON-lines text (one request per line, as the Collector file exporter writes).

{"resourceSpans": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "orders"}}]},
  "scopeSpans": [{"spans": [{"traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "spanId": "00f067aa0ba902b7",
    "parentSpanId": "53995c3f42cd8ad8", "name": "SELECT shop.orders", "kind": 3,
    "startTimeUnixNano": "1760000000016000000", "endTimeUnixNano": "1760000001316000000",
    "attributes": [{"key": "db.system.name", "value": {"stringValue": "postgresql"}}],
    "status": {"code": 0}}]}]}]}

Jaeger JSON (UI download / query API): {"data": [{"traceID", "spans": [...], "processes": {...}}]}, or a single trace object. Times are in µs. A Jaeger error response ("errors": [{"msg": "trace not found"}]) is reported as a failed, uncharged item with that message. CHILD_OF and FOLLOWS_FROM references are understood: FOLLOWS_FROM children count as asynchronous, and extra references become links. Tags (error, span.kind, otel.status_code) and logs (exception events) are mapped. Short IDs with leading zeros stripped are padded back.

Zipkin v2 JSON: a span array (or an array of traces). Times are in µs. remoteEndpoint.serviceName becomes peer.service. Shared B3 single-host server spans that reuse the client's span ID are split into their own span.

Simplified for anything else:

{"trace_id": "checkout-42", "spans": [
  {"span_id": "a1", "service": "frontend", "name": "GET /api/checkout", "kind": "server", "start_ms": 0, "duration_ms": 2350},
  {"span_id": "d1", "parent_span_id": "a1", "service": "orders", "name": "SELECT shop.inventory", "kind": "client",
   "start_ms": 295, "duration_ms": 1450, "status": "error", "status_message": "statement timeout",
   "attributes": {"db.system.name": "postgresql", "db.query.text": "SELECT * FROM inventory WHERE lower(sku) = $1"}}
]}

CamelCase keys (spanId, parentSpanId, startMs, durationMs), end_ms instead of duration_ms, "status": {"code": "ERROR", "message": "..."}, events, links and "follows_from": true are also accepted.

Durations-only mode. If spans have no start_ms, the trace is still analysed, but timing is approximate. Children are assumed sequential when their durations fit inside the parent; otherwise the largest are taken greedily (the "heaviest path"). Timing-dependent detectors (gaps, network/queue time, retry backoff, consumer lag, clock skew, sequential-call detection) are skipped, the critical path is flagged critical_path_approximate: true, and suspect scores are multiplied by 0.75.

A single payload can contain many trace IDs. Spans are grouped by trace ID and each trace is analysed (and billed) separately.

Input reference

Field	Type	Default	Description
traces	array (JSON)	demo trace (prefill)	Required. Trace payloads in any supported format. A bare Zipkin span array can be pasted as is.
baselineTraces	array (JSON)	—	Healthy traces of the same request type, used for regression analysis.
latencySloMs	integer	—	Latency objective. Adds SLO-breach findings and the fleet breach rate.
focusService	string	—	Restricts the analysis window to this service's top-most span and everything it calls. Traces without the service are skipped and not charged.
topRootCauses	integer 1–10	5	Number of ranked suspects per trace.
maxTraces	integer 1–1000	25	Maximum number of traces to analyse, in input order.
thresholds	object	—	Per-run overrides for detector thresholds, e.g. {"slowDbMs": 250}. See Tuning thresholds.
includeWaterfall	boolean	true	Include the ASCII waterfall.
fleetSummary	boolean	true	Add the aggregate item when 2 or more traces are investigated.
failOnSeverity	none/critical/high/medium/low	none	CI gate: fail the run at this finding severity.
disabledRules	array of strings	[]	Rule IDs to skip (see RULES.md). Requires a plan with custom rules.
licenseKey	secret string	—	Optional enterprise or prepaid license key.

How it works

The pipeline is made of pure, separately tested modules: parse/normalize → tree → timing → critical path → errors and patterns → baseline → detectors → ranking → report.

1. Normalize. Every format is converted to one span model with integer nanosecond timestamps, a service name, kind, status, attributes, events and links. Current and legacy OpenTelemetry semantic conventions are both read (http.response.status_code/http.status_code, db.query.text/db.statement, db.system.name/db.system and so on). For grouping and baseline matching, span names are also reduced to a template: UUIDs, long hex IDs, opaque tokens, stand-alone numbers, e-mail addresses and query values become {id}, so GET /users/8812 and GET /users/8813 count as the same operation. Version-like parts (/v1/, HTTP/1.1) are kept. Outputs still show the concrete span name whenever the grouped spans share one.

2. Rebuild the tree. Parent/child links are rebuilt from the IDs, with these repairs:

   • Identical duplicate spans are dropped. Conflicting duplicates are renamed id~dupN and kept.
   • Cycles in parent references are broken.
   • Orphans (parent missing) become separate sub-trees.
   • The primary root is the longest real root. If there is none, it is the longest orphan sub-tree, and the trace is flagged incomplete.
   • Clock skew is corrected Jaeger-style. A child from a different service that starts before or ends after its parent is shifted, together with its sub-tree, so that it sits centred in the parent. Same-service children are never shifted because they share a clock. Asynchronous children (FOLLOWS_FROM, producer→consumer) may outlive their parent and are only corrected if they start before it.
   • CLIENT→SERVER pairs across services are matched for network timing.

3. Timing. A span's self (exclusive) time is its duration minus the interval union of its synchronous children, clipped to the span. With a union instead of a sum, two parallel 100 ms calls inside a 120 ms parent leave 20 ms of self time, not −80 ms. Other measures:

   • Gaps: intervals of a parent that no child covers.
   • Network/queue time: client duration minus server duration, split into before and after the server ran.
   • Parallelism factor: total self time ÷ wall time.
   • Aggregates per service, per operation and per dependency edge.

4. Critical path. A backwards walk from the end of the root. At each step it picks the synchronous child that finished last before the cursor, assigns the gap after that child to the parent, recurses into the child, then moves the cursor to the child's start. Concurrent children hidden behind a longer sibling are never selected. Overflowing children are clipped. The walk is iterative, so very deep traces are safe. Every nanosecond of the root window is attributed to exactly one span, so per-span and per-service shares add up to 100 %.

5. Error origin vs. propagation. Errors are status ERROR, the error tag, or HTTP 5xx. The walk starts at the failed request and, at each level, follows the last-finishing failing child, because earlier failures were usually retried. It passes through intermediate spans that forgot to set an error status. The end of that chain is the origin. Errors above it are propagated, and errors whose parent did not fail are handled. Exception events (type and message) are attached, with secrets redacted.

6. Detectors. 56 rules turn patterns into findings. Each finding names the culprit span, gives a severity, evidence and a recommendation. See RULES.md. Sibling calls are grouped by service, span-name template, remote target and query shape, which is how N+1 loops and retries are recognized even when every call has a different ID in its name.

7. Ranking. Candidates are (service, operation) pairs, scored on four signals between 0 and 1:

   • impact = max(critical-path share ÷ 0.5, error-origin credit). The credit is 1.0 for the origin of the request's failure. When the origin is a timeout on an outgoing call, 80 % of its credit goes to the slowest work under that call, because the timeout is a symptom.
   • evidence = noisy-OR of the strongest finding per rule. The weights are high 0.6, medium 0.35 and low 0.15, each multiplied by the finding's confidence.
   • baseline deviation = log2(slowdown) ÷ 3 (an 8× slowdown gives 1.0), 0.7 for new errors and 0.5 for new operations.
   • score = (0.6·impact + 0.25·evidence + 0.15·deviation), normalized when there is no baseline, then multiplied by a data-quality factor: durations-only 0.75, missing root 0.85, many orphans 0.9. Confidence is high at 0.6 or above, medium at 0.35 or above, and low otherwise.

   The category of a suspect comes from its most specific detectors, such as RCA-DB-001 → database or RCA-RES-001 → resource. Location-only detectors like "dominant span" are used only when nothing specific fired. Each suspect's recommendations combine the detectors' recommendations with category-specific next steps.

8. Baseline comparison (optional). Operations are matched on service and span-name template (see step 1), using robust statistics. A regression needs all three of the following: an incident median at least 2× the baseline median, above 1.2× the baseline p95, and at least 20 ms slower. Also reported: critical-path share shifts of 25 percentage points or more, new operations and services, new errors, call-count increases of 2× or more, and the end-to-end slowdown. A trace is compared (and charged) only if the baseline contains its root operation or at least 20 % of its operations.

Root-cause categories

Category	Typical evidence	Typical next step
database	Slow query, DB error, deadlock or lock timeout	Query plan (EXPLAIN ANALYZE), indexes, locks, DB saturation
code	N+1, sequential awaits, CPU-bound span, uninstrumented gap, exception origin, cache stampede, chatty calls	Batch or eager-load, parallelize, profile, fix the bug
dependency	502/503/504 or UNAVAILABLE from an internal service, slow cache	The callee's own traces, health and deploys; circuit breakers
network	Client ≫ server time, slow DNS/TCP/TLS, connection refused or reset	Proxies and meshes, connection reuse, DNS, network policies
resource	Pool wait or exhaustion, GC, cold start, OOM, throttling (429), consumer lag	Pool and limit sizing, provisioned concurrency, scaling
config	Retries without backoff, retry storms, timeouts at round values, auth failures, cache TTLs	Timeout and retry budgets, backoff with jitter, credentials
external	Slow or failing third-party host (not one of your services)	Provider status, timeouts and circuit breaker, caching, async

Output

One dataset item per trace (a standard Report: score, risk_level, summary, findings, plus everything below in data), and one optional fleet summary item. Abridged example for the retry-storm demo:

{
  "product": "trace-root-cause-analyzer",
  "target": "trace c5c0bd655b7ebfbf636581530c19aaa7",
  "status": "ok",
  "score": 61,
  "risk_level": "medium",
  "summary": {"critical": 0, "high": 11, "medium": 5, "low": 0, "info": 0},
  "findings": [{"rule_id": "RCA-ERR-001", "severity": "high", "kind": "observation",
                "location": "payment › POST /charge [000004ccdd70e019]",
                "description": "The failure originates in payment › POST /charge and propagated through 4 ancestor span(s) to the request.",
                "evidence": {"service": "payment", "operation": "POST /charge",
                             "error": {"message": "service unavailable: payment-db failover in progress", "http_status": 503}}}],
  "data": {
    "trace_id": "c5c0bd655b7ebfbf636581530c19aaa7",
    "input_formats": ["jaeger-json"],
    "root_service": "gateway", "root_operation": "POST /api/checkout",
    "trace_duration_ms": 620.0, "span_count": 25, "services": ["gateway", "checkout", "payment"],
    "request_failed": true, "timing_mode": "timestamps", "parallelism": 1.0,
    "slo": {"slo_ms": 500, "breached": true, "over_by_ms": 120.0},
    "suspected_bottleneck": {"service": "payment", "operation": "POST /charge", "self_ms": 44.0, "share": 0.071},
    "suspected_root_causes": [
      {"rank": 1, "service": "payment", "operation": "POST /charge", "category": "dependency",
       "confidence": "high", "score": 0.977, "critical_path_ms": 396.0, "critical_path_share": 0.639,
       "error_origin": true,
       "reasoning": [
         "Holds 396 ms of exclusive time on the critical path (64% of 620 ms).",
         "Origin of the error that failed the request.",
         "RCA-ERR-002 (high): An error in payment propagated through 3 services: payment → checkout → gateway.",
         "RCA-HTTP-002 (high): payment returned HTTP 503 9 time(s) (on the error chain)."],
       "recommendations": ["The callee (or the proxy in front of it) was unavailable or overloaded: check its health, ..."],
       "signals": {"impact": 1.0, "critical_path_share": 0.639, "error_origin": 1.0, "evidence": 0.922,
                   "baseline_deviation": 0.0, "data_quality_factor": 1.0}},
      {"rank": 2, "service": "checkout", "operation": "POST /charge", "category": "config", "confidence": "medium",
       "reasoning": ["...", "RCA-RETRY-002 (high): 8 retries across 2 retrying call site(s); worst amplification ×9 ..."]}
    ],
    "critical_path": [{"service": "gateway", "operation": "POST /api/checkout", "span_id": "000004ccdd70e001",
                       "duration_ms": 620.0, "self_ms": 20.0, "share": 0.0323}],
    "critical_path_by_service": [{"service": "payment", "ms": 396.0, "share": 0.639}],
    "error_chain": [{"role": "request", "service": "gateway", "operation": "POST /api/checkout", "http_status": 503},
                    {"role": "origin", "service": "payment", "operation": "POST /charge",
                     "message": "service unavailable: payment-db failover in progress", "http_status": 503}],
    "dependency_graph": {"nodes": [...], "edges": [{"source": "checkout", "target": "payment", "calls": 9, "errors": 9, "p50_ms": 52.0}]},
    "baseline": null,
    "recommendations": ["..."],
    "waterfall": "span   duration  |0 ... 620 ms|\ngateway › POST /api/checkout  620 ms |#====...",
    "critical_path_mermaid": "flowchart LR\n  n0[\"gateway<br/>POST /api/checkout<br/>20 ms · 3%\"] ...",
    "data_quality": {"timed": true, "orphan_spans": 0, "clock_skew_adjusted_spans": 0, "caveats": []}
  },
  "disclaimer": "Heuristic analysis of the spans you supplied. ..."
}

Field notes:

• suspected_root_causes: ranked suspects. Each has category, confidence (high/medium/low), a numeric score (0–1), the raw signals, the reasoning (why it was ranked), the detector evidence, the span_ids involved and recommendations.
• critical_path: spans on the path in time order. self_ms is the time attributed to that span on the path, and share is its fraction of trace_duration_ms. Shares add up to 1.
• error_chain: from the failed request (request) through propagated spans to the origin, with messages, status codes and exception type/message (secrets redacted).
• dependency_graph: services, databases, caches, queues and external hosts, with call counts, errors and latency per edge.
• baseline: present when a comparison was made. Includes regressions, critical_path_shifts, new_operations, new_services, new_errors, call_count_increases and the end-to-end comparison.
• data_quality: orphans, roots, skew adjustments, duplicates, cycles, plus human-readable caveats (also listed in errors so they appear in REPORT.md).
• waterfall: a fixed-width timeline. # means on the critical path, = the rest of the span, ! an error. Large traces are trimmed to the root, error chain, critical path and slowest spans.
• Fleet summary item (data.kind = "fleet-summary"): top_root_causes, top_bottlenecks, top_error_origins, operation_latency (p50/p95/p99 per operation across traces), duration_ms percentiles, SLO breach rate, and RCA-FLEET-* findings.
• Failed items (status: "failed") explain why a payload or trace could not be analysed. They are never charged.
• score/risk_level follow the shared scoring of all our Actors (100 = no findings). For traces they are a rough "how many problems" signal; read the suspects for the diagnosis.

INVESTIGATION.md (key-value store) contains the same analysis as a readable report: the fleet table, then per trace the suspects with reasoning and next steps, critical-path table, Mermaid critical path, error chain, waterfall, Mermaid dependency graph and caveats. It renders well on GitHub, GitLab and in most incident tools.

Pricing

Pay per event. You only pay for traces that were actually analysed.

Event	Price	When
trace-investigated	$0.50	Per trace analysed (primary event).
baseline-comparison	$0.25	Per trace that was compared with your baselineTraces. Only charged when the baseline matches the trace (its root operation, or at least 20 % of its operations).
fleet-summary	$0.25	At most once per run, for the aggregate item. See below.

The fleet summary is a separate, paid item. When a run investigates (and charges) 2 or more traces, the Actor adds one extra dataset item that aggregates them: recurring root causes and bottlenecks, latency percentiles per operation, error origins, the SLO breach rate and the RCA-FLEET-* findings. It is charged once, $0.25, no matter how many traces it covers. It is on by default; set fleetSummary: false to skip it and its charge. The per-trace results are the same either way. Runs with a single analysed trace never produce or charge it.

Examples:

Run	Cost
1 trace	1 × $0.50 = $0.50
3 traces, no baseline	3 × $0.50 + $0.25 fleet summary = $1.75 (or $1.50 with fleetSummary: false)
10 traces with a matching baseline	10 × $0.50 + 10 × $0.25 + $0.25 = $7.75

Never charged: unparseable payloads, traces with no usable spans or a zero-duration root, traces skipped by focusService, and traces beyond maxTraces. If your spending limit is reached, the Actor stops before starting work it can't bill: it skips the baseline comparison or the fleet summary first, then stops.

Tuning thresholds

Every number a detector compares against has a default, and you can override any of them for one run with the thresholds input object. For example, a batch job where 300 ms queries are normal and only loops of 10 or more calls are worth flagging:

{"thresholds": {"slowDbMs": 1000, "verySlowDbMs": 5000, "nPlusOneMin": 10}}

Unknown keys and out-of-range values fail the run with an error before anything is charged. ms values are milliseconds; ratios are between 0 and 1; counts are whole numbers. The overrides used are listed in the run's OUTPUT notes.

Key	Default	Rules	What it sets
dominantSpanShare	0.5	RCA-LAT-001	Share of the critical path one span must hold.
dominantSpanHighShare	0.7	RCA-LAT-001	Share at which it becomes high severity.
dominantSpanMinMs	50 ms	RCA-LAT-001/002	Minimum critical-path time of the dominant span or service.
dominantServiceShare	0.75	RCA-LAT-002	Share of the critical path one service must hold.
gapMinMs	50 ms	RCA-LAT-004	Minimum uninstrumented gap.
gapMinFraction	0.2	RCA-LAT-004	Minimum gap as a fraction of the parent span.
cpuLeafMinMs	100 ms	RCA-LAT-005	Minimum self time of a CPU-bound leaf span.
cpuLeafShare	0.25	RCA-LAT-005	Minimum critical-path share of that leaf.
nPlusOneMin	5	RCA-CODE-001, RCA-DB-002	Repeated calls/queries that count as N+1.
nPlusOneHigh	20	RCA-CODE-001, RCA-DB-002	Repeats that make N+1 high severity.
sequentialSavingMinMs	50 ms	RCA-CODE-002	Minimum time saved by parallelizing sequential calls.
sequentialSavingFraction	0.15	RCA-CODE-002	Minimum saving as a fraction of the trace.
fanOutChildren	50	RCA-CODE-003	Children of one span that count as a fan-out.
fanOutConcurrency	25	RCA-CODE-003	Peak concurrent children that count as a fan-out.
fanOutHigh	200	RCA-CODE-003	Children that make a fan-out high severity.
slowDbMs	100 ms	RCA-DB-001	Duration of a slow database call.
verySlowDbMs	500 ms	RCA-DB-001	Duration that makes it high severity.
poolWaitMs	20 ms	RCA-DB-004, RCA-RES-001	Connection-pool wait worth reporting.
networkOverheadMs	50 ms	RCA-NET-001	Client time not spent in the server.
networkOverheadFraction	0.3	RCA-NET-001	That overhead as a fraction of the client span.
connectSlowMs	50 ms	RCA-NET-002	Slow DNS/TCP/TLS connection setup.
externalSlowMs	200 ms	RCA-EXT-001	Slow third-party call.
retryStormAmplification	4	RCA-RETRY-002	Multiplied attempts across nested retry layers.
retryStormTotal	6	RCA-RETRY-002	Retries in one request.
noBackoffGapMs	10 ms	RCA-RETRY-003	Largest gap between attempts that still means no backoff.
timeoutClusterMinMs	250 ms	RCA-TIMEOUT-002	Minimum duration of spans checked for round-number timeouts.
cacheMissRatio	0.5	RCA-CACHE-001	Cache miss ratio.
cacheMinLookups	3	RCA-CACHE-001	Lookups needed before the miss ratio is judged.
slowCacheMs	50 ms	RCA-CACHE-003	Slow cache call.
consumerLagMs	500 ms	RCA-MSG-001	Producer-to-consumer lag.
consumerLagHighMs	5,000 ms	RCA-MSG-001	Lag that makes it high severity.
chattyCalls	20	RCA-TOPO-001	Calls between two services that count as chatty.
largeTraceSpans	1,000	RCA-TOPO-002	Spans that make a trace unusually large.

Rules

56 per-trace rules and 4 fleet rules, with severities, descriptions and recommendations, are listed in RULES.md. Families: LAT latency, CODE call patterns, DB databases, RES resources, NET network, EXT external dependencies, HTTP status codes, RETRY, TIMEOUT, ERR error origin, CACHE, MSG messaging, TOPO topology, DATA data quality, BASE baseline, ROOT top suspect, FLEET aggregate. Rule IDs are stable. Use them in disabledRules.

FAQ

Does it send my traces to an AI model or anywhere else? No. The analysis is deterministic code running inside the Actor, and nothing is sent to other services. Secrets found in error messages and query text are redacted with the same detector our security Actors use. Literal values in SQL are replaced by ?.

How is this different from the waterfall in Jaeger, Tempo or Datadog? Those show the data. This interprets it: it computes the critical path with concurrency handled correctly, separates error origin from propagation, runs 56 failure-pattern detectors, compares with healthy traces and ranks causes with explicit reasoning. You can use the output as a first-pass triage and verify it in your trace UI.

What if my spans have no start times? You get durations-only mode: a self-time / heaviest-path approximation with lower confidence (see "Durations-only mode" above). For the full analysis, export spans with timestamps.

My services' clocks are not in sync. Cross-service skew is detected and corrected Jaeger-style (RCA-DATA-003, data_quality.clock_skew_adjusted_spans). The network/queue split is then omitted for the adjusted calls, because it would be unreliable.

The trace is partial (missing root or dropped spans). The longest available sub-tree is analysed. Orphans and missing roots are reported, and confidence is reduced.

Can I analyse many traces at once? Yes. Pass several payloads, or one payload with many trace IDs. Each trace is a separate item, and with 2 or more you also get the fleet summary. Use maxTraces to cap the cost.

Which semantic conventions are understood? Current and legacy OpenTelemetry HTTP, RPC/gRPC, database, messaging, FaaS and exception conventions, plus common attributes such as peer.service, server.address, cache.hit, http.request.resend_count, db.client.connection.pending_requests and faas.coldstart. Jaeger and Zipkin tags (error, span.kind, http.status_code) are mapped too.

Limitations and responsible use

• Hypotheses, not proof. A trace only shows what was instrumented and sampled. A slow span can be the victim of a noisy neighbour, a saturated node or a lock held by another request that is not in the trace. Confirm with metrics and logs.
• Missing instrumentation shows up as self time or gaps. The Actor reports it (RCA-LAT-004, RCA-DATA-006) but cannot see inside it.
• Detectors are heuristics. Their default thresholds (listed under Tuning thresholds and in RULES.md) were chosen by hand, not fitted to your traffic. Some findings may not matter for your system: raise the threshold with thresholds or turn the rule off with disabledRules. The ranking weights and confidence cut-offs are fixed.
• Durations-only and partial traces produce lower-confidence results, labelled as such.
• Limits: up to 50,000 spans per trace and maxTraces traces per run (plus your plan's per-run target limit). Very large inputs need more memory; the default is 1 GB.
• Only submit traces you are allowed to process. Avoid sending personal data in span attributes. The Actor redacts credentials it recognizes, but it cannot know every kind of sensitive value.

Changelog

See CHANGELOG.md.