Metric Drop & Aggregation Recommender

Find the metrics and labels that nothing reads: no Grafana dashboard, alert, recording rule or logged query. Get ranked drop and aggregation recommendations with series and $/month savings, plus ready-to-apply OpenTelemetry Collector processors, Prometheus metric_relabel_configs, remote-write write_relabel_configs, recording rules and a phased rollout plan. Works like usage-based "adaptive metrics" tools, but runs on your own exports, with no agent and no access to your systems.

Why use it

Most metric bills pay for series nobody looks at: whole exporters nobody charts, pod and instance labels that every panel sums away, histogram buckets when dashboards only show averages. Cardinality reports tell you what is big. This Actor tells you what is unused, which is the part you can safely remove.

Who it's for

• Platform/observability teams cutting Prometheus, Mimir, Grafana Cloud, Chronosphere or other metrics bills
• SREs cleaning up after OTel migrations (duplicate OTel/Prometheus names, old semconv names)
• CI pipelines that should fail when an alert references a metric that no longer exists

How usage is determined

1. Inventory: what you store. Supported formats: metric list JSON, Prometheus /api/v1/series, /api/v1/label/__name__/values and /api/v1/metadata exports, /api/v1/status/tsdb, OTLP/JSON metrics, and Prometheus/OpenMetrics exposition text. Histograms and summaries are grouped into families (_bucket/_sum/_count); their bucket boundaries (le) are kept. Optional per-metric job/jobs (or a job label) tells the Actor which scrape job emits a metric.
2. Usage sources: Grafana dashboards (panels, rows, collapsed rows, legacy rows, templating variables, label_values() / metrics() / query_result() variable queries), Prometheus/Mimir rule files (groups: files, PrometheusRule CRDs, ruler exports) and query lists or Prometheus query logs.
3. Every expression is parsed by a real PromQL parser. Grafana macros ($__rate_interval, $__interval, ${var}, [[var]]) are substituted first, outside string literals. Variables with known options are expanded.
4. For each selector the Actor computes which labels the query depends on, top-down: results shown on a dashboard need all labels, sum by (service) narrows that to service, without (pod) needs everything except pod, on()/group_left() add join keys, label_replace adds its source label, histogram_quantile adds le, and matchers always count.
5. Recording rules are followed transitively. A rule is used only if something used reads its output, and label needs flow through the chain. A metric that only feeds an unused recording rule counts as unused.
6. Unparsable queries fall back to a conservative token scan. Metrics found that way are marked uncertain and are never dropped or aggregated.

PromQL coverage

Selectors (incl. {__name__=~"…"} regex names, UTF-8 quoted names), all matchers, all aggregations (by/without, before or after the argument, topk/quantile/count_values parameters), binary operators with correct precedence, bool, on/ignoring, group_left/group_right, set operators, functions, range vectors, subqueries ([1h:5m]), offset (incl. negative) and @ (incl. start()/end()), comments.

Recommendation types

Type	Rule	What it means
drop_metric	MUO-DROP-001/003/004, MUO-QRY-003	Nothing reads it (or only unused recording rules / stale queries)
drop_family	MUO-DROP-002	≥3 unused metrics share a prefix and nothing under the prefix is used
aggregate_labels	MUO-AGG-001	Every query aggregates these labels away; pre-aggregate with sum/max/min/mean
drop_label_values	MUO-AGG-007	Every query filters to e.g. env="prod"; drop the other values
drop_histogram_buckets	MUO-HIST-001	Only _sum/_count are used
drop_summary_quantiles	MUO-HIST-004	Only _sum/_count of a summary are used
reduce_buckets, exponential_histogram	MUO-HIST-002/003	Only a few quantiles queried (with concrete suggested boundaries) / large classic histogram
increase_interval	MUO-RES-001/002	Only long-range dashboards, or windows ≥ 16× the interval
recording_rule	MUO-QRY-001	Same expensive aggregation computed in many places

Each recommendation carries series before/after, $/month saved (if you gave prices), confidence (high/medium/low), evidence (the queries and their label needs) and risk notes. They are ranked by savings. The aggregation function is derived from how queries aggregate: counters and histograms use sum only; gauges use the operator the queries use, or a name/unit heuristic flagged by MUO-AGG-003. Queries using count, quantile, stddev or non-linear functions before the aggregation block pre-aggregation.

Safety rules

• No usage sources means no drops (MUO-DATA-001).
• Metrics used by alerts are never dropped. Aggregations keep every label the alert expression and its {{ $labels.x }} templates use, and are capped at medium confidence.
• Protected metrics/labels (globs or /regex/; up, scrape_*, ALERTS*, target_info by default) are never touched.
• Uncertain usage (unparsable queries, dynamic metric names) is never dropped. Drops are high confidence only when dashboards and rules/query logs were supplied and everything parsed.
• Counters aggregated across pod/instance get a reset caveat (MUO-AGG-006).
• Prometheus labeldrop is never generated for aggregations: without aggregation it merges distinct series into duplicates. Aggregation goes through the Collector or recording rules.
• Integrity checks: alerts on missing metrics (critical), panels on missing metrics, filters on labels the metric doesn't have, and existing drop rules that remove used metrics or labels.

Existing configuration

Supplied Collector and scrape configurations are read so the Actor does not recommend what is already done and can warn when an existing rule removes something that is used:

• Filter processor OTTL conditions are parsed (and/or/not, name, IsMatch,

  metric.type == METRIC_DATA_TYPE_*
  , unit, resource.attributes["service.name"], HasAttrOnDatapoint) and evaluated per metric. A condition that depends on data-point attributes (e.g. attributes["env"] == "dev") is reported as a partial drop in data_quality instead of being guessed.
• metric_relabel_configs are scoped to their scrape job. With job/jobs in the inventory (or a job label, as in /api/v1/series exports) a job's drop/keep rules only count for the metrics that job emits, and generated relabel rules are only appended to the jobs that emit the affected metrics. Without job information, drop rules count globally and keep lists only when there is a single job.

How to use

In Apify Console

Paste your inventory (e.g. curl prometheus/api/v1/status/tsdb?limit=1000, or a /metrics dump), add dashboard JSON exports, rule files and optionally query logs, enter your prices and run.

Via the API

curl -X POST "https://api.apify.com/v2/acts/michiganpublictech~metric-usage-optimizer/runs?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d @input.json

CI

Set failOnSeverity: "critical" to fail a pipeline when an alert references a metric that is missing from the inventory.

Input

Field	Description
metricInventory	Inventory text/JSON (any supported format, auto-detected)
inventoryFormat	auto (default) or force exposition
dashboards	Grafana dashboard JSON objects (or JSON strings)
ruleFiles	Rule files as YAML strings (or objects)
queries	Query strings or {query, count, lastUsed, source} objects, or Prometheus query-log lines
collectorConfig / scrapeConfig	Existing configuration: honoured, and generated rules are merged into it
analysisName	Label for the analysis
analyses	Batch: list of objects with the per-analysis fields above. Each one is billed separately
costPerActiveSeriesPerMonth, costPerMillionSamples	Your prices (no vendor prices are assumed)
defaultScrapeIntervalSeconds	Interval for sample costs when the inventory has none (60)
minSavingsSeries, minSavingsUsdPerMonth	Hide small recommendations
protectedMetrics, protectedLabels, includeDefaultProtected	Never touch these
staleQueryDays	Ignore queries not run for N days (relative to the newest lastUsed in the input)
highCardinalityThreshold	Cardinality that makes an unused label "high" (100)
outputs	otel-collector, prometheus-relabel, prometheus-remote-write, recording-rules, vendor-neutral
generateConfigs	Turn generated configuration off
failOnSeverity, disabledRules, licenseKey	Licensing & CI (disabling a rule also removes its recommendations)

Output

One dataset item per analysis (the shared Report shape). data contains:

• summary: metrics total/used/unused/uncertain, series and $/month before/after, savings %
• recommendations: the ranked list described above
• usage: per metric, where it is used (dashboards, panels, alerts, recording rules, query executions), labels used/unused and whether it is only used aggregated
• generated: otel_collector_yaml, prometheus_relabel_yaml, prometheus_remote_write_yaml (plus remote_write_recorded: raw → recorded series and the queries to rewrite), recording_rules_yaml, vendor_neutral
• rollout_plan: phases with verification queries to run before and after
• data_quality: notes on partial inventories, parse failures, dynamic variables and so on

Example recommendation:

{"id": "R2", "type": "drop_label_values", "title": "Keep only env=prod for http_server_request_duration_seconds",
 "series_before": 32000, "series_after": 16000, "usd_per_month_saved": 819.2, "confidence": "medium",
 "evidence": ["Every query filters env to ['prod']; 2 values exist."],
 "risk_notes": ["New values of this label (e.g. a new environment) will be dropped too — review when you add one."]}

Generated Collector processors (merged into your config, placed after memory_limiter and before batch):

processors:
  filter/muo_drop:
    error_mode: ignore
    metrics:
      metric:
      - name == "rpc_server_call_duration_seconds"
      - IsMatch(name, "^go_memstats_")
      datapoint:
      - metric.name == "http_server_request_duration_seconds" and not (attributes["env"] == "prod")
  metricstransform/muo_aggregate:
    transforms:
    - include: http_server_requests_total
      match_type: strict
      action: update
      operations:
      - action: aggregate_labels
        label_set: [env, route, service, status_code]
        aggregation_type: sum

Histograms whose buckets are dropped keep _sum/_count: the transform processor extracts them as non-monotonic sums so Prometheus-compatible exporters keep the names your queries use (a monotonic sum would be exported as <name>_sum_total). Generated processors are checked against otelcol-contrib 0.115 and 0.150.

Recording rules follow level:metric:operations, e.g. service:process_cpu_seconds:rate5m: sum by (service) (rate(process_cpu_seconds_total[5m])).

Prometheus remote write (prometheus-remote-write): write_relabel_configs that stop sending unused series to your backend while the local Prometheus keeps everything. With recording-rules selected it also sends only the recorded series of aggregated metrics (raw series stay local); remote_write_recorded lists the dashboard queries to point at the recorded series first. Merged into your remote_write section when the scrape configuration has one. The key-value store also gets OUTPUT, REPORT.md (with an "Estimated savings" summary line, MUO-COST-003) and REPORT.sarif. See RULES.md for all rules.

Rollout plan

0. Shadow & baseline: deploy recording rules and record the "before" verification queries.
1. Drop unused (high confidence).
2. Drop the rest after checking with metric owners.
3. Aggregate labels, one metric at a time, comparing dashboards and alert state.
4. Manual review: bucket layouts (with suggested boundaries that keep your SLO thresholds, plus queries that show where the quantiles fall today), intervals, native histograms.

Pricing

Pay per event:

Event	Price	When
metrics-analyzed	$0.50	One inventory analysed with its usage sources (each item in analyses counts once)
rules-generated	$0.25	Generated configuration contains at least one recommendation

Inventories that cannot be parsed are not charged. Your spending limit is checked before each analysis.

Limitations

• "Unused" only means unused by the sources you supplied. Add query logs to see ad-hoc and API use.
• Savings are estimates. Post-aggregation series = product of kept label cardinalities, capped at today's count.
• Collector metric names: the Actor uses the OTel name when it is known, otherwise the Prometheus name. Check them with the debug exporter.
• Logs, traces and non-Prometheus panels (Loki, SQL…) are skipped. LogQL/MetricsQL extensions are not parsed.

FAQ

Will dropping break my dashboards? Only metrics that no supplied panel, alert, rule or query reads are dropped, and each item comes with verification queries.

Why not labeldrop? It creates duplicate series. See the safety rules.

How is this different from a cardinality analyzer? Cardinality tools rank what is big. This Actor decides what is used, and uses cardinality only to estimate savings.

Changelog

See CHANGELOG.md.