OpenTelemetry Collector Config Optimizer

Paste your OpenTelemetry Collector YAML and get back a score, a prioritised list of reliability, cost and security issues, and an optimized configuration you can review and deploy.

The OpenTelemetry Collector is the most flexible part of a modern observability stack and also one of the easiest to get subtly wrong. A missing memory_limiter means one traffic spike can OOM-kill the Collector. A batch processor in the wrong place wastes CPU on data that is then dropped. retry_on_failure.enabled: false loses data on every backend 503. A single user.id attribute on a metric can multiply your metrics bill. None of these stop the Collector from starting, so they tend to reach production unnoticed.

This Actor runs 56 static checks against a Collector configuration, explains each problem with a precise YAML location (for example service.pipelines.traces.processors[2] or exporters.otlp/backend.sending_queue), and rewrites the configuration with safe, explainable fixes only.

---

Why use it

• Control observability costs. It flags unfiltered high-volume trace pipelines, debug exporters printing every span, very short scrape and collection intervals, high-cardinality attributes on metrics, spanmetrics dimensions such as user.id, logs pipelines without severity filtering, and verbose internal telemetry. Each report includes a separate cost-risk sub-score.
• Improve reliability. It checks memory limiting, batching, processor order, exporter queues and retries, timeouts and persistent queues, so data isn't lost during spikes, backend outages or restarts.
• Harden security. It detects hard-coded API keys and tokens (always redacted in the output), receivers and diagnostic extensions bound to 0.0.0.0, disabled TLS verification for remote backends, unauthenticated OTLP receivers and sensitive attributes passed through without redaction.
• Catch upgrade breakage early. It flags removed and deprecated components and settings (logging exporter, memory_ballast, ballast_size_mib, jaeger exporter, the spanmetrics processor and others) and components that don't exist in the Collector distribution you actually run.
• Get a fix, not only a report. The optimized configuration adds and reorders processors, re-enables queues and retries, migrates logging to debug and moves secrets into ${env:…} references. Every change is listed and linked to the rule that triggered it.

Who it's for

• Platform, SRE and observability teams who run Collectors as agents (DaemonSet or sidecar) or gateways and want a consistent review baseline.
• Teams reviewing pull requests that touch otel-collector.yaml. Run it in CI and fail the build on high-severity findings, with SARIF output for GitHub code scanning.
• Consultancies and vendors who audit customer Collector deployments and need a repeatable, explainable report.
• FinOps / cost owners who want to know which parts of the telemetry pipeline drive ingestion volume.

Features

Area	What is checked
Structure	Invalid YAML, missing service.pipelines, undefined or duplicate component references, unused components, malformed sections, unknown signal types, extra YAML documents
Compatibility	Connectors used on only one side, components missing from the selected distribution, signal mismatches (tail_sampling outside traces, prometheus receiver outside metrics, spanmetrics connector not traces→metrics…), removed and deprecated components and settings
Memory	Missing memory_limiter, not first, missing or zero check_interval, no limit, spike limit missing or too large
Batching	Missing batch, not last, send_batch_max_size < send_batch_size, extreme timeouts and sizes
Processor order	Enrichment after batch (including k8sattributes losing pod association), filtering and sampling after batch, general order deviations
Queue / retry	Retries disabled, queue disabled, extreme queue sizes, non-persistent queues, zero or huge timeouts, infinite retries
Cost	Unsampled high-volume traces, debug exporter with detailed verbosity, scrape interval < 15s, collection interval < 10s, filelog ingesting the Collector's own logs, no log severity filtering, detailed internal metrics, debug internal logs
Cardinality	High-cardinality attributes added to metrics (attributes, resource, transform/OTTL, groupbyattrs), spanmetrics/servicegraph dimensions, metricstransform without aggregation, delta backends without cumulativetodelta
Security	0.0.0.0 receivers, insecure TLS to remote endpoints, hard-coded credentials, exposed pprof/zpages/health_check, unauthenticated OTLP receivers, sensitive attributes without a redaction processor
Vendors	Datadog API key, Honeycomb x-honeycomb-team / x-honeycomb-dataset headers, Grafana Cloud authentication and OTLP/HTTP gateway

The full list, with severities, recommendations and links to opentelemetry.io and the component READMEs, is in RULES.md.

It also understands:

• Component IDs with names (batch/traces, otlp/backend) and named pipelines (traces/sampled).
• Connectors as both exporter and receiver, including signal-pair validation and downstream pipelines. For example, sampling in a pipeline fed by a forward connector counts for the upstream pipeline.
• Extensions and service.extensions, plus service.telemetry.
• Environment-variable references ${env:VAR}, ${VAR} and ${env:VAR:-default}. Values that use them are never reported as hard-coded secrets, and unknown hosts behind env vars are not flagged.
• Multi-document YAML: the first document is analysed and the others are reported as ignored.

How to use

In Apify Console

1. Open the Actor and paste your configuration into Collector configuration (YAML), or add one or more Configuration URLs. GitHub blob links are converted to raw links automatically.
2. Pick the Collector distribution you run: otelcol-contrib (default), otelcol (core) or otelcol-k8s.
3. Click Start. Results appear in the Output tab:
   • Overview: one row per configuration with score, risk level, finding counts, cost risk and pipelines.
   • Findings: one row per finding with severity, rule, YAML location, recommendation and evidence.
   • Optimized configuration: the rewritten YAML, the list of changes and suggestions, and the environment variables you need to set.

The key-value store also contains OUTPUT (run summary), REPORT.md (a human-readable report) and REPORT.sarif (for GitHub code scanning, on plans that include SARIF).

Via the API (curl)

curl -s -X POST \
  "https://api.apify.com/v2/acts/michiganpublictech~otel-config-optimizer/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @- <<'EOF'
{
  "configUrls": ["https://github.com/acme/platform/blob/main/deploy/otel-collector.yaml"],
  "distribution": "k8s",
  "optimize": true,
  "minSeverity": "low"
}
EOF

To send a local file, encode it as a JSON string:

jq -n --rawfile cfg otel-collector.yaml '{configYaml: $cfg, distribution: "contrib"}' |
  curl -s -X POST -H "Content-Type: application/json" -d @- \
  "https://api.apify.com/v2/acts/michiganpublictech~otel-config-optimizer/run-sync-get-dataset-items?token=$APIFY_TOKEN" |
  jq '.[0] | {score, risk_level, summary, cost: .data.cost_risk}'

Python (apify-client)

from pathlib import Path
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("michiganpublictech/otel-config-optimizer").call(
    run_input={
        "configYaml": Path("otel-collector.yaml").read_text(),
        "distribution": "contrib",
        "optimize": True,
    }
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["target"], item["score"], item["risk_level"])
    for f in item["findings"]:
        print(f"  [{f['severity']}] {f['rule_id']} {f['location']}: {f['title']}")
    if item["data"]["optimized_configuration"]:
        Path("otel-collector.optimized.yaml").write_text(item["data"]["optimized_configuration"])
        for var in item["data"]["required_env_vars"]:
            print("  set env var:", var["name"], "-", var["description"])

GitHub Actions: lint otel-collector.yaml on pull requests

name: otel-collector-lint
on:
  pull_request:
    paths: ["deploy/otel-collector.yaml"]

jobs:
  lint:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write   # upload SARIF to code scanning
    steps:
      - uses: actions/checkout@v4
      - name: Analyse Collector config
        env:
          APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }}
        run: |
          jq -n --rawfile cfg deploy/otel-collector.yaml \
            '{configYaml: $cfg, distribution: "k8s", failOnSeverity: "high", optimize: true}' > input.json
          # Start the run and wait for it to finish (non-zero status when the gate fails)
          RUN=$(curl -s -X POST -H "Content-Type: application/json" -d @input.json \
            "https://api.apify.com/v2/acts/michiganpublictech~otel-config-optimizer/runs?token=$APIFY_TOKEN&waitForFinish=240")
          STORE=$(echo "$RUN" | jq -r .data.defaultKeyValueStoreId)
          STATUS=$(echo "$RUN" | jq -r .data.status)
          curl -s "https://api.apify.com/v2/key-value-stores/$STORE/records/REPORT.sarif?token=$APIFY_TOKEN" -o otel.sarif
          curl -s "https://api.apify.com/v2/key-value-stores/$STORE/records/REPORT.md?token=$APIFY_TOKEN" >> "$GITHUB_STEP_SUMMARY"
          echo "status=$STATUS" >> "$GITHUB_OUTPUT"
          test "$STATUS" = "SUCCEEDED"
      - name: Upload SARIF
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: otel.sarif

With failOnSeverity: "high" the run fails when any finding is high or critical, so the job fails too. The SARIF upload annotates the pull request, and REPORT.md is added to the job summary.

Input reference

Field	Type	Default	Description
configYaml	string (textarea)	demo config (prefill)	Full Collector YAML. Use ${env:VAR} for secrets.
configUrls	array of strings	[]	Public http(s) URLs of Collector YAML files. GitHub blob URLs are converted to raw. Private/internal addresses are refused (SSRF protection). Max 2 MiB per file.
distribution	contrib | core | k8s	contrib	The Collector build that runs the config. Used for component-availability checks.
optimize	boolean	true	Produce the optimized configuration (charged only when at least one change is applied).
minSeverity	info | low | medium | high | critical	info	Hide findings below this severity. The score always considers every finding.
failOnSeverity	none | critical | high | medium | low	none	CI gate: fail the run when a finding is at or above this severity.
disabledRules	array of strings	[]	Rule IDs to skip (e.g. OTEL-QUEUE-004). Requires a plan with custom rules.
licenseKey	secret string	—	Optional enterprise / prepaid license key.

At least one of configYaml or configUrls is required. Each configuration produces one dataset item and is billed separately. Configurations that can't be fetched or parsed are reported with errors populated and are not charged.

Output

One dataset item per configuration. Example (shortened) for the demo configuration:

{
  "product": "otel-config-optimizer",
  "product_version": "0.1.0",
  "target": "configYaml (pasted)",
  "generated_at": "2026-10-11T17:24:33.890845Z",
  "score": 62,
  "risk_level": "medium",
  "summary": { "critical": 0, "high": 4, "medium": 4, "low": 1, "info": 3 },
  "findings": [
    {
      "rule_id": "OTEL-MEM-001",
      "title": "Pipeline has no memory_limiter processor",
      "severity": "high",
      "kind": "configuration",
      "category": "memory",
      "location": "service.pipelines.metrics.processors",
      "description": null,
      "evidence": { "pipeline": "metrics" },
      "recommendation": "Add a `memory_limiter` processor as the first processor of every pipeline that receives data, e.g. `check_interval: 1s`, `limit_percentage: 80`, `spike_limit_percentage: 25`. ...",
      "references": ["https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md"],
      "confidence": 1.0,
      "fingerprint": "f4800f86ce4fbd0d"
    }
  ],
  "data": {
    "distribution": "contrib",
    "distribution_name": "OpenTelemetry Collector Contrib (otelcol-contrib)",
    "pipelines": {
      "traces":  { "signal": "traces",  "receivers": ["otlp"], "processors": ["batch"], "exporters": ["otlphttp"] },
      "metrics": { "signal": "metrics", "receivers": ["otlp"], "processors": ["batch", "attributes/enrich"], "exporters": ["otlphttp", "logging"] }
    },
    "component_counts": { "receivers": 1, "processors": 2, "exporters": 2, "connectors": 0, "extensions": 0 },
    "env_vars_referenced": [],
    "cost_risk": { "score": 67, "level": "medium", "drivers": ["OTEL-CARD-001", "OTEL-COST-001", "OTEL-COST-002"] },
    "optimized_configuration": "# Optimized by OpenTelemetry Collector Config Optimizer v0.1.0 (Apify Actor).\n# Generated automatically - REVIEW BEFORE DEPLOYING. ...\nreceivers:\n  ...",
    "optimization_changes": [
      { "rule_id": "OTEL-QUEUE-001", "path": "exporters.otlphttp.retry_on_failure.enabled", "description": "Re-enabled retry_on_failure on exporter 'otlphttp'.", "applied": true },
      { "rule_id": "OTEL-MEM-001", "path": "service.pipelines.traces.processors[0]", "description": "Added 'memory_limiter' as the first processor.", "applied": true },
      { "rule_id": "OTEL-SEC-001", "path": "receivers.otlp.protocols.grpc.endpoint", "description": "Suggestion (not applied): bind to '${env:MY_POD_IP}:4317' ...", "applied": false }
    ],
    "required_env_vars": [],
    "notes": []
  },
  "errors": [],
  "disclaimer": "Static analysis of a Collector configuration file. ..."
}

Field notes:

• score runs from 0 to 100, where 100 means no issues. It's deterministic and shared by all our Actors: each severity has a base weight (critical 25, high 12, medium 5, low 1.5, info 0) and repeated findings of the same severity decay geometrically. risk_level is derived from the score. Unparsable configurations score 0.
• data.cost_risk is a separate heuristic (100 = no cost drivers found). Each finding in the cost or cardinality categories, plus a missing batch processor, subtracts high 25 / medium 15 / low 7 / info 3. The level is low at 80 or above, medium at 50 or above, and high below 50. Use it to rank configurations by likely ingestion cost; it isn't a dollar estimate.
• data.optimization_changes lists applied changes (applied: true) and suggestions (applied: false) that you should apply by hand.
• data.required_env_vars lists environment variables introduced by the optimizer when it replaced hard-coded secrets. The original values are never included.
• fingerprint is stable per rule and location, so you can track findings across runs.

The demo configuration's optimized output looks like this:

# Optimized by OpenTelemetry Collector Config Optimizer v0.1.0 (Apify Actor).
# Generated automatically - REVIEW BEFORE DEPLOYING. Validate with `otelcol validate --config=<file>`.
# Comments and anchors from the original file are not preserved.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317       # reported as a suggestion, never changed automatically
processors:
  batch: {}
  attributes/enrich:
    actions:
      - key: user.id                 # still reported (OTEL-CARD-001): the optimizer never removes your logic
        from_attribute: enduser.id
        action: insert
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
exporters:
  otlphttp:
    endpoint: https://otlp.example.com
    retry_on_failure:
      enabled: true
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, attributes/enrich, batch]
      exporters: [otlphttp, debug]

(The real output uses block-style lists and has no inline comments. They're added here for explanation.)

The recommended processor order

The optimizer and the OTEL-MEM-002, OTEL-BATCH-002 and OTEL-ORDER-* rules follow the order recommended in the Collector processor documentation:

1. memory_limiter: first, so back-pressure is applied before any other processor allocates memory for incoming data.
2. Enrichment (k8sattributes, resourcedetection, resource): early, while the original client connection context is still available. k8sattributes associates data with pods by the sender's IP, which is lost after batch.
3. Filtering and transformation (filter, transform, attributes, redaction, span, metricstransform, cumulativetodelta, …): drop and reshape data before spending resources on it, and redact sensitive data before it can leave the Collector.
4. Sampling (tail_sampling, probabilistic_sampler, groupbytrace): after filtering, so sampling decisions only consider data you would keep.
5. batch: last, so it batches only data that will actually be exported.

Processors this Actor doesn't recognise keep their position relative to the processor they followed. Processors of the same stage keep their original relative order.

What the optimizer changes, and what it never changes

It may change:

Change	Rule(s)
Add memory_limiter (check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25) to pipelines fed by receivers, and move it first	OTEL-MEM-001, OTEL-MEM-002
Fill missing check_interval, limits or spike limits on existing memory limiters	OTEL-MEM-003/004/005
Add batch to pipelines that export to real exporters, and move it last	OTEL-BATCH-001, OTEL-BATCH-002
Raise an invalid send_batch_max_size to send_batch_size	OTEL-BATCH-003
Reorder the remaining processors by stage (stable)	OTEL-ORDER-*
Re-enable sending_queue / retry_on_failure that were explicitly disabled	OTEL-QUEUE-001/002
Replace the removed logging exporter with debug (mapping loglevel to verbosity) and update pipelines	OTEL-COMPAT-004
Replace hard-coded credentials with ${env:<NAME>} placeholders (keeping Bearer/Basic prefixes) and list the variables	OTEL-SEC-003
Define memory_limiter/batch processors that pipelines reference but that are missing	OTEL-STRUCT-003

It never:

• deletes a component or a pipeline (unused components are reported, not removed),
• changes bind addresses. 0.0.0.0 receivers get a suggestion such as ${env:MY_POD_IP}:4317 because the right value depends on how the Collector is deployed,
• changes TLS settings, endpoints, sampling policies, filters, OTTL statements or attribute actions,
• migrates components that need a design decision (jaeger exporter → OTLP, spanmetrics processor → connector, memory_ballast → GOMEMLIMIT). These remain findings with clear recommendations,
• keeps YAML comments and anchors. The output is regenerated from the parsed document, which the header comment states.

The optimizer is idempotent: running it on its own output produces the same YAML and no further changes. When no safe change applies, no optimized configuration is produced and none is charged.

Always review the optimized configuration and validate it with otelcol validate --config=<file> (or your distribution's binary) before deploying.

Distributions and vendor checks

The distribution input selects a profile of the components that ship in each upstream build (opentelemetry-collector-releases):

Value	Binary	Notes
contrib (default)	otelcol-contrib	Almost every community component. Components unknown to this Actor are reported as info ("cannot verify").
core	otelcol	Core components only. k8sattributes, tail_sampling, transform, file_storage, filelog and others are reported as unavailable (high).
k8s	otelcol-k8s	Curated for Kubernetes (k8sattributes, kubeletstats, filelog, tail_sampling, …).

The component lists are approximate and are reviewed with each release of this Actor. If you build a custom Collector with ocb, use contrib and treat OTEL-COMPAT-006 findings as a reminder to check your manifest.

Vendor checks run automatically when the configuration targets a known backend:

• Datadog: the datadog exporter needs api.key.
• Honeycomb: OTLP exporters to *.honeycomb.io need x-honeycomb-team, and metrics need x-honeycomb-dataset.
• Grafana Cloud: OTLP exporters to *.grafana.net need Basic auth (header or basicauth extension), and the OTLP gateway should be used with otlphttp.

Both registries are pluggable, so more distributions (for example ADOT, Splunk, Elastic or vendor builds) and vendor checks can be added without changing the rest of the analysis. See the roadmap.

Pricing

This Actor uses pay-per-event pricing. You pay only for results, and the Apify platform usage is included.

Event	Price	When it is charged
config-analyzed	$0.05	Once per configuration analysed with the full rule set
optimized-config	$0.05	Once per configuration for which an optimized configuration was generated with at least one applied change

• Configurations that can't be fetched or parsed are free. They're still listed with the reason.
• optimize: false (or a plan without optimized output) never charges optimized-config.
• Apify's maximum cost per run (spending cap, set in the run options) is respected. The Actor checks the remaining budget before each configuration and stops cleanly when the cap is reached. If the budget covers the analysis but not the optimized configuration, it publishes the analysis without the optimized config and doesn't charge for it.

Example: analysing 10 configurations with optimization, where 7 need changes, costs 10 × $0.05 + 7 × $0.05 = $0.85.

Plans and licensing

Apify Store users get the full product on pay-per-event pricing: all rules, the optimized configuration, SARIF export and the CI gate. License keys (licenseKey) unlock more: Pro adds disabling rules (disabledRules) and up to 500 configurations per run, Team adds bulk runs (2,000 per run), and Enterprise adds white-label reports. Prepaid/invoiced keys skip pay-per-event charges. Without a key and outside Apify (local evaluation) the Actor runs on the Community plan, which has no optimized configuration and a maximum of 3 configurations per run. Missing features never fail a run: they are skipped with a note in the output. See ../../docs/licensing.md for details.

Limitations

• Static analysis only. The Actor reads the configuration file; it doesn't connect to your Collector or backends. Findings describe what the YAML declares (kind: configuration). Actual throughput, memory use and cardinality depend on your traffic.
• Environment variables aren't resolved. Values such as ${env:ENDPOINT} are treated as unknown, so checks that need the value (for example TLS to a remote host) are skipped for them.
• Configuration merging isn't modelled. If you run the Collector with several --config files or --set overrides, analyse the merged result (otelcol print-initial-config / your deployment's rendered config). Helm values.yaml files must be rendered first.
• Component knowledge is curated. Signal support, defaults and deprecations are tracked for well-known components. Unknown and custom components get structural checks only.
• Heuristics can misfire. Cardinality and secret detection use attribute names and patterns. Use disabledRules (on plans with custom rules) or minSeverity to tune noise.
• The output loses YAML comments and anchors. The optimized YAML is regenerated from the parsed document.
• Only the first YAML document is analysed in multi-document files.

FAQ

Will the optimized config work as a drop-in replacement? It's designed to, and only safe edits are made, but you must review it and validate it with otelcol validate. Component behaviour can differ between Collector versions.

Does my configuration or secrets leave Apify? The configuration is processed inside the Actor run. Hard-coded secrets are never written to the output: findings show a redacted value (for example ghp_…aa (40 chars)), and the optimized configuration contains ${env:…} placeholders instead. Rotate any credential you find this way, because it's already exposed wherever the file lives.

Why does a config with no errors still get findings? Many findings are about cost and resilience, not validity. A config can be valid and still drop data on the first backend outage.

Why isn't my 0.0.0.0 endpoint changed automatically? Changing a bind address can break traffic depending on Service, hostNetwork or sidecar topology. You get a concrete suggestion instead.

Can I analyse Helm charts or the OpenTelemetry Operator OpenTelemetryCollector CR? Render them first and pass the config: section as YAML. Native support is on the roadmap.

How do I silence a rule? Use disabledRules (plans with custom rules) or raise minSeverity. Rule IDs are stable and never reused.

How is the score calculated? See the Output field notes above. The same formula is used by all our Actors, so scores are comparable across runs and safe to use as CI gates.

Roadmap

• More distribution profiles: AWS Distro for OpenTelemetry (ADOT), Splunk, Elastic, Grafana Alloy (OTel mode), Dynatrace and other vendor builds.
• More vendor checks: New Relic, Dynatrace, Splunk Observability, Elastic, Google Cloud, AWS (X-Ray/EMF), Lightstep/ServiceNow.
• Native input for Helm values and OpenTelemetry Operator OpenTelemetryCollector resources.
• Collector-version awareness (collectorVersion input) for precise deprecation timelines.
• Agent/gateway topology checks (for example loadbalancing exporter in front of tail_sampling).
• Diff-friendly output (unified diff between original and optimized config).

Changelog

See CHANGELOG.md.