# OpenTelemetry Demo Telemetry Generator (`michiganpublictech/otel-demo-generator`) Actor

Generate realistic, correlated OpenTelemetry traces, metrics and logs for demos, POCs and trainings. Scripted incident scenarios streamed over OTLP to any observability backend.

- **URL**: https://apify.com/michiganpublictech/otel-demo-generator.md
- **Developed by:** [Justin Hagerty](https://apify.com/michiganpublictech) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$20.00 / 1,000 telemetry minute delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## OpenTelemetry Demo Telemetry Generator

**Realistic, correlated OpenTelemetry traces, metrics and logs for a whole microservice system — with a scripted incident, a clear root cause and a recovery — streamed over OTLP/HTTP to the observability backend you want to show off.**

Great observability demos need great data. A real demo app (Kubernetes cluster, ten services, a load generator, a way to break things on cue) takes days to set up, costs money to keep running, and never fails at the moment you need it to. Dashboards built on an empty sandbox look sad, and "imagine this chart had data" doesn't close deals.

This Actor simulates an entire microservice system — storefront, API gateway, checkout, payments, databases, caches, Kafka, third-party APIs — and plays a **scenario**: a healthy baseline, then (optionally) an incident such as a slow payment provider, a bad deployment, a missing database index or a memory leak, followed by recovery. Everything is generated with current OpenTelemetry semantic conventions and is **correlated**: logs carry `trace_id`/`span_id`, metrics and traces share resources, the incident shows up consistently in all three signals. You get a **demo guide** with the exact UTC timeline and the clicks to make.

Nothing to deploy. Point it at an OTLP endpoint, pick a scenario, start.

***

### Who it's for

- **Sales engineers and solution architects** running live demos and POCs of an observability platform.
- **Observability vendors' partners and consultants** who need a repeatable, prospect-branded demo environment (`companyName`, `serviceNamespace`).
- **Trainers and workshop hosts** teaching distributed tracing, RED metrics, SLOs or incident response — every attendee sees the same incident at the same time (`seed`).
- **Platform teams** building dashboards, alerts and SLOs before real services are instrumented, or smoke-testing a telemetry pipeline (Collector → backend) with realistic, non-sensitive data.

### What you get

- **Correlated traces, metrics and logs** from 7–10 services in Java, Go, Python, Node.js and .NET, with databases, Redis, Kafka/RabbitMQ, Elasticsearch and third-party APIs.
- **9 scripted scenarios** with talk tracks: latency incident, bad deployment and rollback, slow queries, memory leak with OOMKilled restarts, cache stampede, third-party outage with circuit breaker, traffic surge with autoscaling, credential stuffing — plus a healthy baseline.
- **4 industries**: e-commerce, fintech, B2B SaaS and logistics, each with its own service graph and business attributes.
- **11 backend presets**: generic OTLP, your own Collector, Honeycomb, Grafana Cloud, New Relic, Elastic, Dynatrace, SigNoz, Jaeger, Uptrace, Axiom.
- **Two modes**: *live* (real time, data every ~15 s — for the demo itself) and *backfill* (the last N minutes, sent in seconds — to prefill dashboards before the meeting).
- **A demo guide** (`DEMO_GUIDE.md`) with the UTC timeline of every phase, the walkthrough in order, talk-track bullets and suggested queries for your backend.
- **Delivery-health report**: every run checks your destination configuration before sending and reports authentication failures, wrong paths, throttling, partial rejections and more — with a score you can gate CI on.
- **Free dry runs** that store sample OTLP/JSON payloads and volume estimates without sending anything.
- **Deterministic**: the same `seed` reproduces the same telemetry, byte for byte.

### Scenario catalog

The incident starts at `incidentStartMinute` (default 5) and lasts `incidentDurationMinutes` (default 8), including onset and recovery. `incidentSeverity` (low / medium / high) scales the effects. Service names below are for the e-commerce flavor; other industries map the same roles to their own services.

| Scenario (`scenario`) | What happens | What the demo shows | Talk track in one line |
|---|---|---|---|
| `healthy-baseline` | Steady traffic with gentle sinusoidal variation, ~0.5% errors | Service map, RED metrics, SLO building, trace ↔ log pivots | "Everything instrumented, everything correlated — this is your baseline." |
| `checkout-latency-incident` (default) | External `payment-provider` gets ~10x slower: timeouts, retries, ERROR logs; then recovers | Alert → service map → slow traces → the long CLIENT span to the third party → correlated logs → recovery | "From symptom to root cause in four clicks, without leaving the tool." |
| `bad-deployment-rollback` | `cart` 2.4.1 → 2.5.0 rolls out (`service.version` flips), errors and latency regress, rollback | Deployment markers, compare by `service.version`, exceptions with stack traces | "Know which deploy broke production — and prove the rollback fixed it." |
| `database-slow-queries` | A migration drops an index: one query becomes 20–50x slower, the connection pool saturates, upstream latency rises | `db.client.operation.duration` by collection, `db.query.text` on slow spans, pool saturation metrics | "Many endpoints slow at once? Look for the shared resource." |
| `memory-leak-oom` | `recommendation` (JVM) heap grows linearly, GC pauses rise, OOMKilled restarts with error bursts, hotfix | Saw-tooth `jvm.memory.used`, `k8s.container.restarts`, new `service.instance.id` per restart, GC pauses | "Metrics, logs and traces on one timeline tell the whole OOM story." |
| `cache-stampede` | Redis is flushed: hit ratio collapses, the database is hammered, then the cache warms up | `app.cache.hits/misses`, DB load spike, traces with cache miss → SELECT | "A thundering herd, caught in the act." |
| `third-party-outage` | `shipping-carrier` returns 503s, the circuit breaker opens, fallbacks are served, then it recovers | Errors isolated to one `server.address`, `app.circuit_breaker.state`, fallback spans, low user-facing error rate | "Resilience patterns working — and visible." |
| `traffic-surge` | Flash sale: 4–5x traffic, the autoscaler adds pods, Kafka consumer lag climbs then drains | Throughput, new `service.instance.id`s, CPU per pod, `kafka.consumer_group.lag` | "Capacity planning with real numbers." |
| `security-anomaly` | Credential stuffing: hundreds of 401s on the login route from a few IPs (documentation ranges), then rate limiting (429) | Status codes by route, group by `client.address`, WARN logs for forensics | "Observability data is security data too." |

### Industries (topologies)

| `industry` | Services | Infrastructure & third parties | Business attributes |
|---|---|---|---|
| `ecommerce` (default) | frontend (Node.js), api-gateway (Go), product-catalog (Java), cart (.NET), checkout (Java), payment (Go), inventory (Go), shipping (Python), recommendation (Java), email-worker (Python) | PostgreSQL ×3, Redis, Kafka; payment provider, shipping carrier, email delivery | `app.customer.tier`, `app.region`, `app.cart.items`, `app.order.value_usd`, feature flag |
| `fintech` | web-app, api-gateway, accounts, transactions, ledger, fraud-scoring, kyc, notifications-worker | PostgreSQL, Redis, Kafka; card network, KYC provider, notification provider | tier, region, transfer value, feature flag |
| `saas` | web-app, api-gateway, auth, projects, search, billing, webhooks-worker | PostgreSQL, Redis, RabbitMQ, Elasticsearch; payments API, email provider, webhook delivery | tier, region, subscription value, feature flag |
| `logistics` | portal, api-gateway, orders, routing, fleet-tracking, warehouse, notifications | PostgreSQL, Redis, Kafka; carrier API, maps API, notification provider | tier, region, order value, feature flag |

All service and dependency names are generic (third-party hosts use the reserved `.example` domain). Each run's `data.services` lists the services with language, version and replica count.

### Supported backends

Presets fill in the default endpoint and signal paths, check the headers the vendor expects, and pick a protocol/temporality the vendor accepts. **Endpoints and header names change — confirm them in your vendor's OTLP documentation.** The Actor never ships credentials; you provide them in `headers` (stored as a secret).

| `vendorPreset` | Endpoint (`otlpEndpoint`) | Authentication header(s) | Notes |
|---|---|---|---|
| `generic-otlp` | your base URL, e.g. `https://otlp.example.com` | whatever your backend needs | `/v1/traces`, `/v1/metrics`, `/v1/logs` are appended |
| `otel-collector` | e.g. `https://collector.example.com:4318` | only if your receiver uses an authenticator | OTLP/HTTP is port **4318** (4317 is gRPC, not supported). Must be reachable from the internet |
| `honeycomb` | default `https://api.honeycomb.io` (EU: `https://api.eu1.honeycomb.io`) | `x-honeycomb-team: <ingest key>`; optional `x-honeycomb-dataset` | dataset header is needed for metrics |
| `grafana-cloud` | your stack's OTLP gateway, e.g. `https://otlp-gateway-prod-us-east-0.grafana.net/otlp` | Basic auth: instance ID + access-policy token → use `basicAuthUser` / `basicAuthPassword` | traces → Tempo, metrics → Mimir, logs → Loki |
| `new-relic` | default `https://otlp.nr-data.net` (EU: `https://otlp.eu01.nr-data.net`) | `api-key: <license key>` | protobuf only (JSON is switched automatically) |
| `elastic` | your APM / managed OTLP endpoint | `Authorization: ApiKey <key>` (or `Bearer <secret token>`) | protobuf |
| `dynatrace` | `https://{environment-id}.live.dynatrace.com/api/v2/otlp` | `Authorization: Api-Token <token>` (trace, metric and log ingest scopes) | protobuf only; **delta** metrics (selected automatically) |
| `signoz` | default `https://ingest.us.signoz.cloud:443` (also `eu`, `in`) | `signoz-ingestion-key: <key>` | self-hosted: use `otel-collector` |
| `jaeger` | your Jaeger OTLP/HTTP receiver, e.g. `https://jaeger.example.com:4318` | usually none | traces only (metrics and logs disabled automatically) |
| `uptrace` | default `https://api.uptrace.dev` | `uptrace-dsn: <project DSN>` | self-hosted: set `otlpEndpoint` |
| `axiom` | default `https://api.axiom.co` | `Authorization: Bearer <token>`, `X-Axiom-Dataset: <dataset>` | check metrics support for your dataset type |

Only public endpoints can be reached: the Actor runs in the cloud and refuses private, loopback and link-local addresses (SSRF protection). Allowed ports: 80, 443, 4318, 8080, 8443.

### Quick start

#### 1. Preview for free (dry run)

The input form is prefilled with a **dry run**: click **Start**. Nothing is sent and nothing is charged. You get:

- `DEMO_GUIDE.md` — the presenter guide for the scenario;
- `SAMPLE_TRACES.json`, `SAMPLE_METRICS.json`, `SAMPLE_LOGS.json` — one minute of OTLP/JSON payloads from the incident peak (first 25 traces with their logs, plus the full metrics snapshot);
- a dataset item with per-minute volumes, estimated totals and payload sizes for the whole run.

#### 2. Honeycomb

```json
{
  "vendorPreset": "honeycomb",
  "headers": "x-honeycomb-team: <your ingest key>\nx-honeycomb-dataset: demo-metrics",
  "confirmAuthorization": true,
  "scenario": "checkout-latency-incident",
  "mode": "live",
  "durationMinutes": 20,
  "incidentStartMinute": 8
}
```

Start it ~5 minutes before the demo; the guide tells you exactly when the incident begins.

#### 3. Grafana Cloud

```json
{
  "vendorPreset": "grafana-cloud",
  "otlpEndpoint": "https://otlp-gateway-prod-us-east-0.grafana.net/otlp",
  "basicAuthUser": "<instance ID>",
  "basicAuthPassword": "<access policy token>",
  "confirmAuthorization": true,
  "scenario": "database-slow-queries",
  "mode": "backfill",
  "durationMinutes": 30
}
```

Copy the endpoint and instance ID from your stack's *OpenTelemetry* tile. `basicAuthPassword` is a secret field.

#### 4. New Relic

```json
{
  "vendorPreset": "new-relic",
  "headers": "api-key: <license key>",
  "confirmAuthorization": true,
  "scenario": "bad-deployment-rollback",
  "industry": "fintech"
}
```

EU accounts: add `"otlpEndpoint": "https://otlp.eu01.nr-data.net"`.

#### 5. Your own OpenTelemetry Collector

```json
{
  "vendorPreset": "otel-collector",
  "otlpEndpoint": "https://collector.example.com:4318",
  "headers": "Authorization: Bearer <token>",
  "confirmAuthorization": true,
  "scenario": "traffic-surge"
}
```

A minimal receiver for it (put TLS and an authenticator in front of anything public):

```yaml
extensions:
  bearertokenauth:
    token: ${env:DEMO_INGEST_TOKEN}
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: bearertokenauth
```

The Collector then fans the data out to any backend — a good way to demo several tools side by side.

### Modes and timing

| | `live` (default) | `backfill` |
|---|---|---|
| Timestamps | from "now" forward, minute by minute | the last `durationMinutes`, ending at the start of the current minute |
| Sending | paced in real time: spans and logs every ~15 s, metrics every 60 s (the SDK default export interval) | as fast as the backend accepts |
| Run time | ≈ `durationMinutes` (+ up to ~30 s) | seconds to a few minutes |
| Best for | the live demo itself; workshops | prefilling dashboards before a meeting; building and testing alerts |

**Run timeout:** a live run lasts as long as `durationMinutes` (up to 120). Make sure the run's timeout (Run options → Timeout) is longer than that; the default run options are set to more than 2 hours. Backfills are quick.

Some backends reject metric points that are too old; the Actor flags long backfills for those vendors (`OTD-CFG-008`).

### Input reference

| Field | Type | Default | Description |
|---|---|---|---|
| `dryRun` | boolean | `false` (prefilled `true`) | Generate but don't send; store samples, guide and estimates. Free. |
| `vendorPreset` | enum | `generic-otlp` | Backend preset (see table above). |
| `otlpEndpoint` | string | preset default | OTLP/HTTP base URL. Required for presets without a default. `https://` recommended. |
| `headers` | secret text | — | One header per line, `Name: value` (a JSON object also works). Never written to output or logs. |
| `basicAuthUser` / `basicAuthPassword` | string / secret | — | Builds `Authorization: Basic …` (Grafana Cloud). |
| `protocol` | `http/protobuf` | `http/json` | `http/protobuf` | OTLP/HTTP encoding. JSON uses hex trace/span IDs per the OTLP spec. |
| `compression` | `gzip` | `none` | `gzip` | Request compression. |
| `confirmAuthorization` | boolean | `false` | Required (`true`) when sending: you own or are authorized to use the endpoint. |
| `scenario` | enum | `checkout-latency-incident` | See the scenario catalog. |
| `industry` | `ecommerce` | `fintech` | `saas` | `logistics` | `ecommerce` | System topology. |
| `incidentStartMinute` | integer | 5 (fitted for runs < 15 min) | When the incident begins. |
| `incidentDurationMinutes` | integer | 8 (fitted for runs < 15 min) | Incident length incl. onset and recovery. Start + duration must be ≤ `durationMinutes`. |
| `incidentSeverity` | `low` | `medium` | `high` | `medium` | Scales the effects. |
| `mode` | `live` | `backfill` | `live` | See *Modes and timing*. |
| `durationMinutes` | integer 1–120 | 15 | Simulated minutes. |
| `trafficLevel` | `low` | `medium` | `high` | `medium` | ≈ 30 / 120 / 600 request traces per minute (hard cap 1,500/min during surges). |
| `signals` | array of `traces`, `metrics`, `logs` | all | Signals to send. |
| `seed` | integer | random | Reproducible telemetry; the seed used is reported in `data.seed`. |
| `environment` | string | `demo` | `deployment.environment.name`. |
| `serviceNamespace` | string | industry name | `service.namespace` and `k8s.namespace.name`. |
| `companyName` | string | — | Adds `app.company` and shows in the guide. |
| `extraResourceAttributes` | object | — | Up to 20 extra resource attributes. |
| `maxBatchSize` | integer 100–10,000 | 2,000 | Max items per request (requests also stay under ~3 MB). |
| `metricsTemporality` | `auto` | `cumulative` | `delta` | `auto` | Per-vendor default (delta for Dynatrace). |
| `failOnSeverity` | `none` … `low` | `none` | CI gate on delivery health, e.g. `critical` fails the run when the key is rejected. |
| `disabledRules` | array | — | Rule IDs to skip (Pro plan and above). |
| `licenseKey` | secret | — | Enterprise / prepaid keys. |

Invalid input stops the run before anything is sent, with a message saying what to fix (e.g. an incident that doesn't fit in the run, a missing endpoint, or a missing authorization confirmation).

### Output

#### Dataset (one item per run)

The standard report shared by all our Actors; `score` is the **delivery-health score** (100 = everything delivered, no configuration problems). Example (shortened) for a 15-minute Honeycomb backfill:

```json
{
  "product": "otel-demo-generator",
  "product_version": "0.1.0",
  "target": "https://api.honeycomb.io",
  "status": "ok",
  "score": 98,
  "risk_level": "low",
  "summary": { "critical": 0, "high": 0, "medium": 0, "low": 1, "info": 0 },
  "findings": [
    {
      "rule_id": "OTD-CFG-010",
      "title": "Recommended vendor header missing",
      "severity": "low",
      "kind": "configuration",
      "location": "headers.x-honeycomb-dataset",
      "evidence": { "preset": "honeycomb", "missing": "x-honeycomb-dataset" },
      "recommendation": "Add the header in 'headers'. Confirm its meaning in the vendor's OTLP documentation."
    }
  ],
  "data": {
    "mode": "backfill",
    "dry_run": false,
    "scenario": "checkout-latency-incident",
    "scenario_title": "Checkout latency incident (slow third-party API)",
    "industry": "ecommerce",
    "vendor_preset": "honeycomb",
    "protocol": "http/protobuf",
    "endpoint": "https://api.honeycomb.io",
    "header_names": ["x-honeycomb-team"],
    "seed": 42,
    "duration_minutes": 15,
    "minutes_delivered": 15,
    "time_range": { "start": "2026-10-03T03:45:00Z", "end": "2026-10-03T04:00:00Z" },
    "timeline": [
      { "phase": "baseline", "start": "2026-10-03 03:45:00 UTC", "end": "2026-10-03 03:50:00 UTC", "incident": false,
        "description": "Normal traffic and healthy dependencies." },
      { "phase": "degradation", "start": "2026-10-03 03:50:00 UTC", "end": "2026-10-03 03:52:00 UTC", "incident": true,
        "description": "payment-provider starts responding slowly; checkout p99 climbs." }
    ],
    "services": [{ "name": "frontend", "language": "nodejs", "version": "3.12.0", "replicas": 3, "operations": 6 }],
    "counts": {
      "traces": { "generated": 2167, "sent": 2167, "accepted": 2167, "rejected": 0 },
      "spans": { "generated": 20538, "sent": 20538, "accepted": 20538, "rejected": 0, "failed": 0 },
      "metric_data_points": { "generated": 4693, "sent": 4693, "accepted": 4693, "rejected": 0, "failed": 0 },
      "log_records": { "generated": 2252, "sent": 2252, "accepted": 2252, "rejected": 0, "failed": 0 }
    },
    "http_status_counts": { "200": 45 },
    "delivery": { "traces": { "requests": 15, "bytes": 1089311, "retries": 0, "disabled": null } },
    "demo_guide_key": "DEMO_GUIDE.md",
    "stopped_early": null,
    "notes": []
  },
  "errors": [],
  "disclaimer": "Synthetic telemetry for demos, trainings and testing. It contains no real user data. ..."
}
```

`counts.traces.accepted` is derived from span acceptance (spans of one trace can travel in different requests). `endpoint` and `target` never include credentials or query strings; `header_names` lists names only, never values. Dry runs add `data.dry_run_preview` (sample minute, per-minute averages, estimated totals and payload bytes).

Dataset views in Console: **Overview** (scenario, backend, minutes delivered, counts, score), **Scenario timeline** and **Delivery findings**.

#### Key-value store

| Record | Content |
|---|---|
| `DEMO_GUIDE.md` | Presenter guide (every run, including dry runs) |
| `SAMPLE_TRACES.json`, `SAMPLE_METRICS.json`, `SAMPLE_LOGS.json` | OTLP/JSON samples (dry runs only) |
| `OUTPUT` | Run summary: plan, score, billing |
| `REPORT.md` / `REPORT.sarif` | Delivery-health report (SARIF on plans that include it) |

#### Demo guide excerpt

```markdown
## Demo guide — Checkout latency incident (slow third-party API)

| Phase | Start (UTC) | End (UTC) | What happens |
|---|---|---|---|
| baseline | 03:45 | 03:50 | Normal traffic and healthy dependencies. |
| degradation | 03:50 | 03:52 | payment-provider starts responding slowly; checkout p99 climbs. |
| incident | 03:52 | 03:56 | payment-provider latency is ~10x baseline: timeouts and retries; ... |
| recovery | 03:56 | 03:58 | payment-provider recovers; latency drains back to normal. |

### Walkthrough (in this order)

1. Open the alert / SLO view: checkout error budget burning since the degradation phase.
2. Open the service map: checkout and upstream api-gateway/frontend show elevated latency and errors.
3. Chart p99 of http.server.request.duration for checkout: compare baseline vs incident.
4. Filter traces: service.name = checkout AND duration > 2s. Open one.
5. In the waterfall, the CLIENT span to payment-provider (server.address) takes seconds and is retried.
6. Click through to correlated logs (same trace_id): timeout ERRORs with exception.type.

#### Honeycomb
- **Latency of the focus service:** `VISUALIZE HEATMAP(duration_ms), P99(duration_ms) WHERE service.name = checkout GROUP BY name`
```

The guide also contains the root cause (for the presenter), talk-track bullets, generic queries that work anywhere, metrics to chart, and vendor-specific queries for Honeycomb, Grafana Cloud (TraceQL, PromQL, LogQL), New Relic (NRQL), Elastic (KQL) and Jaeger. Verify query syntax against your backend version: attribute mapping can differ by ingestion settings.

### Telemetry details

**Traces.** Root SERVER spans at the edge service with `client.address`, `user_agent.original` and business attributes; CLIENT/SERVER pairs for HTTP (`http.request.method`, `http.route`, `url.path`, `url.full`, `http.response.status_code`, `server.address`) and gRPC (`rpc.system`, `rpc.service`, `rpc.method`, `rpc.grpc.status_code`); database CLIENT spans (`db.system.name` + legacy `db.system`, `db.namespace`, `db.operation.name`, `db.collection.name`, parameterized `db.query.text` — no literals, no PII); Redis spans with `app.cache.hit`; Kafka/RabbitMQ PRODUCER spans and CONSUMER spans in a new trace **linked** to the producer; INTERNAL spans for business logic and circuit breakers. Failures set span status ERROR, `error.type` and an `exception` event with `exception.type`, `exception.message` and a language-appropriate `exception.stacktrace`. Retries carry `http.request.resend_count`.

**Metrics** (cumulative by default, 60 s interval, consistent start times; delta for delta-only backends): `http.server.request.duration` and `http.client.request.duration` (histograms with the semantic-convention buckets), `rpc.server.duration`, `rpc.client.duration`, `db.client.operation.duration`, `db.client.connection.count` / `.max` / `.pending_requests`, `messaging.client.sent.messages`, `messaging.client.consumed.messages`, `messaging.process.duration`, `kafka.consumer_group.lag` or `rabbitmq.message.current`, `process.cpu.utilization`, `process.memory.usage`, `jvm.memory.used`, `jvm.memory.limit`, `jvm.gc.duration`, `k8s.container.restarts`, `app.cache.hits`, `app.cache.misses`, `app.circuit_breaker.state`, `app.business.transactions`.

**Logs.** Access logs at the edge, business events, WARN/ERROR logs for retries, timeouts, failed logins, rate limits, slow queries and fallbacks — all with `trace_id`/`span_id` — plus lifecycle and scenario events (deployments, rollbacks, autoscaling, OOMKilled, circuit breaker state changes) tagged with `event.name`.

**Resources.** `service.name`, `service.namespace`, `service.version`, `service.instance.id`, `deployment.environment.name` (and legacy `deployment.environment`), `telemetry.sdk.*`, `process.runtime.*`, `k8s.cluster/namespace/deployment/pod/container/node.*`, `host.name`, `cloud.provider/platform/region/availability_zone`, `app.company` (optional), your extra attributes, and **`app.synthetic = true`** on every resource so synthetic data is always easy to filter out or delete.

**Cardinality budget.** Attribute values are bounded: route templates instead of raw paths on metrics, no user IDs, a handful of pods per service (autoscaling adds a few), three customer tiers, four regions. A 15-minute medium run produces roughly 300 metric data points per minute — comfortably small for any backend.

### Delivery health and rules

Before sending, the Actor checks the destination configuration (`OTD-CFG-*`, kind `configuration`); while sending, it records what the backend answered (`OTD-DLV-*`, kind `runtime`). Examples: credentials over plain HTTP, a missing vendor header, JSON for a protobuf-only vendor, the gRPC port 4317, rejected credentials (critical), wrong path (404), partial-success rejections, throttling (429), server errors, payload too large (413), timeouts. A clean run has no findings and a score of 100. The full catalog — 22 rules with recommendations — is in **[RULES.md](RULES.md)**.

How sending works: requests are batched (≤ `maxBatchSize` items, ≤ ~3 MB), gzip-compressed, and retried with exponential backoff and jitter on 429/502/503/504 and network errors, honouring `Retry-After` (capped at 30 s). 400/401/403/404/413/415 are not retried; a 413 batch is split and resent. On 401/403/404 the Actor stops sending that signal immediately instead of hammering your endpoint. `partial_success` responses are parsed (protobuf and JSON) and counted as rejected.

### API and automation

#### Run and get the report (curl)

```bash
jq -n --arg key "$HONEYCOMB_KEY" '{
  vendorPreset: "honeycomb",
  headers: ("x-honeycomb-team: " + $key),
  confirmAuthorization: true,
  scenario: "cache-stampede",
  mode: "backfill",
  durationMinutes: 30,
  seed: 2026
}' | curl -s -X POST -H "Content-Type: application/json" -d @- \
  "https://api.apify.com/v2/acts/michiganpublictech~otel-demo-generator/run-sync-get-dataset-items?token=$APIFY_TOKEN" |
  jq '.[0] | {score, minutes: .data.minutes_delivered, counts: .data.counts}'
```

(`run-sync-*` endpoints wait up to 300 s — use them for backfills; start live runs asynchronously with `/runs`.)

#### Python (apify-client)

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("michiganpublictech/otel-demo-generator").call(
    run_input={
        "vendorPreset": "new-relic",
        "headers": f"api-key: {os.environ['NEW_RELIC_LICENSE_KEY']}",
        "confirmAuthorization": True,
        "scenario": "memory-leak-oom",
        "industry": "saas",
        "mode": "backfill",
        "durationMinutes": 45,
        "incidentStartMinute": 15,
        "incidentDurationMinutes": 20,
    },
    timeout_secs=900,
)
item = next(client.dataset(run["defaultDatasetId"]).iterate_items())
print(item["score"], item["data"]["minutes_delivered"], item["data"]["http_status_counts"])
guide = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("DEMO_GUIDE.md")["value"]
print(guide)
```

#### Keep a demo environment fresh every day

1. In Apify Console, create a **Task** for this Actor with your backend settings, `mode: "backfill"`, `durationMinutes: 120` and `scenario: "healthy-baseline"` (or rotate scenarios with several tasks).
2. Create a **Schedule** (e.g. cron `0 7 * * 1-5`, your timezone) that runs the task.
3. Set `failOnSeverity: "critical"` and a run-failure notification, so you learn about an expired API key before the demo, not during it.

Two hours of baseline at medium traffic costs 120 × $0.02 = $2.40 per refresh.

### Pricing

Pay-per-event: you pay only for minutes your backend accepted. Apify platform usage is included.

| Event | Price | When it is charged |
|---|---|---|
| `telemetry-minute` | **$0.02** | Once per simulated minute of telemetry that was delivered (at least one export request for that minute was accepted) |

- **Dry runs are free.** So are minutes that could not be delivered (rejected credentials, unreachable endpoint…).
- If authentication fails on the first minute, the run stops immediately and **nothing is charged**.
- Apify's **maximum cost per run** (spending cap in the run options) is respected: the Actor checks the remaining budget before generating each minute and stops cleanly when it's reached (noted in `data.stopped_early`).

Examples:

| Usage | Minutes | Cost |
|---|---|---|
| 15-minute live demo | 15 | **$0.30** |
| 1-hour workshop | 60 | $1.20 |
| Maximum run (2 hours) | 120 | $2.40 |
| Daily 2-hour backfill, 22 workdays | 2,640 | $52.80 / month |

The price doesn't depend on `trafficLevel` — but your backend's ingest bill does (see *Responsible use*).

### Plans and licensing

Apify Store users get everything on pay-per-event, including SARIF export and the CI gate. License keys (`licenseKey`) unlock more: **Pro** adds disabling rules (`disabledRules`), **Enterprise** adds white-label reports, and prepaid/invoiced keys skip pay-per-event charges. Missing features never fail a run: they're skipped with a note. See [docs/licensing.md](../../docs/licensing.md).

### Responsible use

- **Only send to endpoints you own or are authorized to use.** The Actor requires `confirmAuthorization: true` and never sends anything in dry runs. Don't send demo data to production tenants without the owner's consent.
- **The data is synthetic.** No real users, customers or credentials; IP addresses come from the documentation ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24); hostnames use `.example`. Every resource carries `app.synthetic = true`.
- **Ingest costs are on the backend side.** Approximate volume per simulated minute (e-commerce, healthy baseline; incidents add a little, `traffic-surge` up to ~4.5x):

| `trafficLevel` | Traces / min (incl. async) | Spans / min | Log records / min | Metric points / min | Protobuf / min | gzip on the wire / min | 1 hour (spans, uncompressed) |
|---|---|---|---|---|---|---|---|
| `low` | ~37 | ~360 | ~37 | ~270 | ~240 KB | ~37 KB | ~22k spans, ~14 MB |
| `medium` | ~145 | ~1,400 | ~150 | ~290 | ~620 KB | ~95 KB | ~84k spans, ~37 MB |
| `high` | ~720 | ~6,900 | ~720 | ~310 | ~2.5 MB | ~360 KB | ~410k spans, ~150 MB |

A dry run reports exact per-minute numbers and an estimated total for your settings.

### Limitations

- **OTLP/HTTP only** (protobuf or JSON). OTLP/gRPC (port 4317) isn't supported.
- **Public endpoints only.** Private, loopback and internal addresses are blocked on purpose; expose your Collector behind TLS + auth, or use a tunnel.
- **Simulated, not instrumented.** The service graph, latencies and failure modes are modeled to look realistic; they're not captured from running software. Profiles, browser RUM and exemplars aren't generated.
- **One metric export per minute.** Metrics have 60-second resolution (like the SDK default); spans and logs arrive every ~15 s in live mode.
- **Backfill is limited to 120 minutes**, and some backends reject old metric points (flagged per vendor).
- **Presets reflect the vendors' public docs at the time of release.** Confirm endpoints, header names and limits in your vendor's documentation; use `generic-otlp` for anything else.
- **Live runs occupy the run for the whole duration.** Set the run timeout longer than `durationMinutes`.

### FAQ

**Will the data look real in my backend's service map?**
Yes. Every call is a CLIENT span in the caller and a SERVER span in the callee with matching `server.address`, so service maps and dependency graphs build automatically. Databases, caches, brokers and third-party APIs appear as dependencies.

**Can I show the same incident twice?**
Yes — reuse the `seed`. Timestamps move with the run's start time, everything else (IDs, values, which requests fail) is identical.

**How do I hide or delete the demo data afterwards?**
Filter on `app.synthetic = true`, `deployment.environment.name` or `service.namespace`. Use a dedicated dataset / environment / tenant for demos where your backend supports it.

**Why does my Honeycomb run report "Recommended vendor header missing"?**
Metrics need a dataset (`x-honeycomb-dataset`). Add it, or disable metrics in `signals`.

**The run failed with "Backend rejected the credentials". Was I charged?**
No. Minutes are charged only when the backend accepted data, and the Actor stops at the first authentication failure.

**Can I send to two backends at once?**
Send to your own OpenTelemetry Collector and fan out from there, or start two runs with the same `seed`.

**Is JSON or protobuf better?**
Protobuf: smaller, supported everywhere. JSON is useful to inspect payloads or for receivers that only accept JSON.

**Does it work with Datadog / Splunk / others?**
Use `generic-otlp` with the vendor's OTLP/HTTP intake URL and headers, or route through your Collector. More presets are on the roadmap.

### Roadmap

- More presets (Datadog OTLP intake, Splunk Observability, Google Cloud, AWS, Azure Monitor, Logz.io, Coralogix, Lightstep/ServiceNow).
- Custom scenarios: compose effects and phases in the input.
- More industries (gaming, healthcare, media streaming) and bring-your-own topology (YAML).
- Exemplars on histograms, span metrics, browser/RUM and profiling signals.
- OTLP/gRPC support.
- Multiple concurrent incidents and "noisy neighbour" background anomalies for AIOps demos.

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

# Changelog

This Actor's version history is a separate document: https://apify.com/michiganpublictech/otel-demo-generator/changelog.md

# Actor input Schema

## `dryRun` (type: `boolean`):

Generate the scenario but do not send it. Stores one minute of sample OTLP/JSON payloads (SAMPLE_TRACES.json, SAMPLE_METRICS.json, SAMPLE_LOGS.json), the demo guide (DEMO_GUIDE.md) and volume estimates in the key-value store. Dry runs are never charged and ignore the endpoint and credentials. Uncheck to send data.

## `vendorPreset` (type: `string`):

Where the telemetry goes. Presets fill in the default endpoint and signal paths and check the authentication headers each vendor expects. Endpoints and header names follow the vendors' public docs — confirm them in your vendor's OTLP documentation.

## `otlpEndpoint` (type: `string`):

Base URL of the OTLP/HTTP receiver; the Actor appends /v1/traces, /v1/metrics and /v1/logs. Required unless the preset has a default (Honeycomb, New Relic, SigNoz, Uptrace, Axiom). Examples: https://otlp.example.com, https://collector.example.com:4318, https://otlp-gateway-prod-us-east-0.grafana.net/otlp, https://{env-id}.live.dynatrace.com/api/v2/otlp. Use https://. Must be reachable from the public internet; ports 80, 443, 4318, 8080 and 8443 are allowed.

## `headers` (type: `string`):

Authentication and routing headers, one per line as 'Name: value' (a JSON object also works). Examples: 'x-honeycomb-team: <key>', 'api-key: <license key>', 'Authorization: Api-Token <token>', 'Authorization: Bearer <token>'. Stored encrypted; values are never written to the output or logs.

## `basicAuthUser` (type: `string`):

Builds 'Authorization: Basic base64(user:password)' for you — e.g. the Grafana Cloud instance ID. Replaces any Authorization header above.

## `basicAuthPassword` (type: `string`):

Password or token for Basic auth (e.g. a Grafana Cloud access policy token with write scopes).

## `protocol` (type: `string`):

OTLP/HTTP encoding. Binary protobuf is supported everywhere; JSON (with hex trace/span IDs per the OTLP spec) is switched to protobuf automatically for vendors that only accept protobuf.

## `compression` (type: `string`):

gzip reduces egress and request size by ~5-8x.

## `confirmAuthorization` (type: `boolean`):

Required when sending (not for dry runs). The data is synthetic, but it counts toward the backend's ingest volume and bill. Don't send it to production tenants without the owner's consent.

## `scenario` (type: `string`):

The story the telemetry tells: a healthy baseline or an incident with a clear root cause and recovery. See the README for what each demo shows.

## `industry` (type: `string`):

Which microservice system to simulate. Each has 7-10 services in Java, Go, Python, Node.js and .NET plus databases, caches, a message broker and third-party APIs.

## `incidentStartMinute` (type: `integer`):

Minute (from the start of the run) when the incident begins. Default 5 (fitted automatically for runs shorter than 15 minutes). Start + duration must fit inside the run.

## `incidentDurationMinutes` (type: `integer`):

How long the incident lasts, including onset and recovery phases. Default 8.

## `incidentSeverity` (type: `string`):

Scales latency, error rates and other effects of the incident.

## `mode` (type: `string`):

live: stream in real time for durationMinutes (data every ~15 s) — best during a live demo; the run lasts as long as durationMinutes, so set the run timeout higher. backfill: timestamps cover the last durationMinutes ending now and are sent as fast as possible — best to prefill dashboards before a demo.

## `durationMinutes` (type: `integer`):

Simulated minutes of telemetry (1-120). Each delivered minute is one 'telemetry-minute' event.

## `trafficLevel` (type: `string`):

Request traces per minute at baseline: low ≈ 30, medium ≈ 120, high ≈ 600 (hard cap 1,500/min during surges). Async consumer traces come on top. See the README for ingest volume per level.

## `signals` (type: `array`):

Which signals to send. Default: all three (traces, metrics, logs).

## `seed` (type: `integer`):

Same seed + same inputs = the same telemetry (IDs, values, timings relative to the start). Leave empty for a random seed; the seed used is reported in the output.

## `environment` (type: `string`):

deployment.environment.name on every resource.

## `serviceNamespace` (type: `string`):

service.namespace and k8s.namespace.name. Defaults to the industry (shop, bank, saas, logistics).

## `companyName` (type: `string`):

Adds app.company to every resource and shows in the demo guide — handy for prospect-branded POCs.

## `extraResourceAttributes` (type: `object`):

Up to 20 additional resource attributes (string, number or boolean values), e.g. {"team": "presales", "demo.id": "acme-poc"}. service.name, service.instance.id and app.synthetic cannot be overridden.

## `maxBatchSize` (type: `integer`):

Upper bound on spans / data points / log records per OTLP request (requests are also kept under ~3 MB). Lower it if the backend answers 413.

## `metricsTemporality` (type: `string`):

auto: the preset's preference (delta for Dynatrace, cumulative elsewhere). In delta mode up/down counters are sent as gauges.

## `failOnSeverity` (type: `string`):

CI gate on delivery health: mark the run as FAILED when any finding (e.g. authentication rejected = critical, endpoint not found = high) is at or above this severity. 'none' disables the gate.

## `disabledRules` (type: `array`):

Rule IDs to skip, e.g. OTD-CFG-010 (see RULES.md). Requires the custom_rules feature (Pro plan and above); otherwise ignored with a note.

## `licenseKey` (type: `string`):

Optional. Enterprise / prepaid license key. Store users don't need one.

## Actor input object example

```json
{
  "dryRun": true,
  "vendorPreset": "generic-otlp",
  "protocol": "http/protobuf",
  "compression": "gzip",
  "confirmAuthorization": false,
  "scenario": "checkout-latency-incident",
  "industry": "ecommerce",
  "incidentSeverity": "medium",
  "mode": "live",
  "durationMinutes": 15,
  "trafficLevel": "medium",
  "signals": [
    "traces",
    "metrics",
    "logs"
  ],
  "environment": "demo",
  "maxBatchSize": 2000,
  "metricsTemporality": "auto",
  "failOnSeverity": "none"
}
```

# Actor output Schema

## `results` (type: `string`):

The run report: scenario timeline, counts per signal, HTTP statuses and delivery findings.

## `demoGuide` (type: `string`):

Presenter guide (Markdown): timeline in UTC, walkthrough, talk track and suggested queries (DEMO_GUIDE.md).

## `summary` (type: `string`):

Plan, score and billing summary for the run (OUTPUT record).

## `reportMarkdown` (type: `string`):

Human-readable delivery-health report (REPORT.md).

## `reportSarif` (type: `string`):

SARIF 2.1.0 report of delivery findings for CI tools (REPORT.sarif).

## `sampleTraces` (type: `string`):

One minute of sample OTLP/JSON traces, written by dry runs only (SAMPLE_TRACES.json).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dryRun": true,
    "vendorPreset": "generic-otlp",
    "scenario": "checkout-latency-incident",
    "industry": "ecommerce",
    "mode": "live",
    "durationMinutes": 15,
    "trafficLevel": "medium"
};

// Run the Actor and wait for it to finish
const run = await client.actor("michiganpublictech/otel-demo-generator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "dryRun": True,
    "vendorPreset": "generic-otlp",
    "scenario": "checkout-latency-incident",
    "industry": "ecommerce",
    "mode": "live",
    "durationMinutes": 15,
    "trafficLevel": "medium",
}

# Run the Actor and wait for it to finish
run = client.actor("michiganpublictech/otel-demo-generator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dryRun": true,
  "vendorPreset": "generic-otlp",
  "scenario": "checkout-latency-incident",
  "industry": "ecommerce",
  "mode": "live",
  "durationMinutes": 15,
  "trafficLevel": "medium"
}' |
apify call michiganpublictech/otel-demo-generator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,michiganpublictech/otel-demo-generator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bqBB273VGkr8hHHyM/builds/es00Teo04N8MtYWUa/openapi.json