# LLM Latency & Cost Monitor (`constant_quadruped/llm-latency-cost-monitor`) Actor Measures cost, speed, and latency across simulated LLM provider APIs for routing optimization. - **URL**: https://apify.com/constant\_quadruped/llm-latency-cost-monitor.md - **Developed by:** [CQ](https://apify.com/constant_quadruped) (community) - **Categories:** AI, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## LLM Latency & Cost Monitor Probes real LLM provider APIs to measure **live wall-clock latency and reachability**, and pairs each result with a **versioned reference pricing config** so you can compare providers for routing / cost optimization. This actor performs genuine network I/O. Every latency value is a real `performance.now()` measurement of an actual HTTPS round-trip to the provider's API host. Nothing is randomized, simulated, or hardcoded. Pricing is the one explicitly non-live field, and it is clearly labelled as a reference config in every output record (see [Pricing](#pricing-is-a-reference-config-not-a-live-feed)). ### What it does For each requested provider, the actor runs one of two **real** probes: 1. **Authenticated probe** — if an API key is available for that provider (from the input or environment), it sends a minimal real chat/completions request (`max_tokens: 1`, prompt `"ping"`) and measures end-to-end latency plus the real HTTP status code. 2. **Unauthenticated probe** — if no key is present, it still does a real probe: a plain HTTPS `GET` to the provider's API host (e.g. `/v1/models`). This returns a real round-trip time and typically an HTTP `401`/`403` (expected without a key), which confirms reachability and measures network latency without spending any API credits. Probes for all providers run concurrently. Each result is pushed to the dataset, and a roll-up is written to the `SUMMARY` key in the default key-value store. #### Supported providers `OpenAI`, `Anthropic`, `Google`, `DeepSeek`, `Mistral`, `Groq`. An unrecognized provider name is **not** fabricated — it is returned with `status: "unsupported-provider"`, `latencyMs: null`, and an explanatory `error`. ### Input All inputs are optional. | Field | Type | Default | Description | |-------|------|---------|-------------| | `providers` | array of strings | `["OpenAI","Anthropic","Google","DeepSeek"]` | Providers to probe. Supported names listed above. Blank/whitespace entries are ignored. | | `apiKeys` | object (secret) | _none_ | Map of provider name to API key, e.g. `{"OpenAI":"sk-..."}`. When a key is present for a provider, that provider gets an **authenticated** probe (a real minimal chat request). Marked secret. | | `models` | object | _none_ | Map of provider name to model id to probe, e.g. `{"OpenAI":"gpt-4o-mini"}`. Only used for authenticated probes; otherwise a cheap default model per provider is used. | | `timeoutMs` | integer | `15000` | Hard per-probe timeout in milliseconds. A probe that exceeds it is reported with `status: "timeout"` and its real elapsed time. Non-positive / non-numeric values fall back to the default. | #### API keys via environment variables Keys can also be supplied as environment variables instead of (or in addition to) `apiKeys`: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` (or `GEMINI_API_KEY`), `DEEPSEEK_API_KEY`, `MISTRAL_API_KEY`, `GROQ_API_KEY`. The output records where each key came from in the `keySource` field (`input.apiKeys`, `input.`, or `env.`). #### Example input ```json { "providers": ["OpenAI", "Anthropic", "Groq"], "timeoutMs": 12000, "apiKeys": { "OpenAI": "sk-..." } } ```` With this input, OpenAI is probed authenticated (real chat request), while Anthropic and Groq are probed unauthenticated (real host latency). ### Output One dataset record per provider. Example (unauthenticated probe, no key supplied): ```json { "provider": "OpenAI", "latencyMs": 224, "inputTokenCostPerMillion": 0.15, "outputTokenCostPerMillion": 0.60, "pricingSource": "reference-config", "pricingConfigVersion": "2026-06-21", "pricingModel": "gpt-4o-mini", "timestamp": "2026-06-24T11:44:56.647Z", "status": "reachable-unauthorized", "probeMode": "unauthenticated", "probedModel": null, "httpStatus": 401, "keySource": null, "error": null } ``` #### Field reference | Field | Meaning | |-------|---------| | `provider` | Provider name that was probed. | | `latencyMs` | Real measured wall-clock latency of the probe, in ms (`performance.now`). `null` only when no probe could run (e.g. unsupported provider). | | `inputTokenCostPerMillion` / `outputTokenCostPerMillion` | USD per 1,000,000 tokens, from the reference pricing config. `null` if no price entry exists for the provider. | | `pricingSource` | Always `"reference-config"`. | | `pricingConfigVersion` | Date/version of the pricing reference used. | | `pricingModel` | The specific model the reference price applies to. | | `timestamp` | ISO 8601 time the measurement was taken. | | `status` | Derived from the real result: `online` (2xx), `reachable-unauthorized` (401/403), `rate-limited` (429), `server-error` (5xx), `http-`, `timeout`, `error`, or `unsupported-provider`. | | `probeMode` | `authenticated`, `unauthenticated`, or `none` (unsupported provider). | | `probedModel` | Model id used for an authenticated probe; `null` for unauthenticated probes. | | `httpStatus` | Raw HTTP status code, or `null` on connection error/timeout. | | `keySource` | Where the key came from, or `null` if unauthenticated. | | `error` | Error message on failure, else `null`. | A `SUMMARY` object is also written to the default key-value store with `probedAt`, `providerCount`, `pricingConfigVersion`, `anyAuthenticated`, and the full `records` array. ### Pricing is a reference config, not a live feed Token prices come from a maintained, versioned config (`src/pricing.js`), captured from each provider's public pricing page on the `lastUpdated` date. **They are not fetched live.** Every record carries `pricingSource: "reference-config"` and `pricingConfigVersion` so you always know the prices are reference values, not a real-time quote. Update `src/pricing.js` (and its `version`/`lastUpdated`) when provider prices change. Latency, by contrast, is always measured live on every run. ### Limitations - **Pricing is not live** — see above. Verify against the provider's current pricing page before making cost decisions. - **Latency depends on where the actor runs** — measured round-trip time reflects the network path between the Apify run's region and the provider, not the latency you would see from your own infrastructure. - **Unauthenticated probes measure host reachability + network latency, not model inference time.** Only an authenticated probe (with a key) reflects real end-to-end chat-completion latency, and even then it is a single 1-token request, not a benchmark. - **Single sample per run** — each provider is probed once. For trend data, schedule the actor and aggregate runs. - A probe failure (timeout, DNS, connection reset) is reported honestly in `status`/`error`; it does not abort the other providers. ### Local development ```bash npm install ## write an input first: ## storage/key_value_stores/default/INPUT.json npx apify run --purge ``` Requires Node.js 18+ (uses the global `fetch` and `AbortController`). # Actor input Schema ## `providers` (type: `array`): LLM providers to probe. Supported: OpenAI, Anthropic, Google, DeepSeek, Mistral, Groq. ## `apiKeys` (type: `object`): Optional map of provider name to API key, e.g. {"OpenAI": "sk-..."}. If provided, the actor fires a minimal REAL chat request and measures end-to-end latency. If omitted, it falls back to a real unauthenticated HTTPS latency probe against each provider host. Keys are also read from env vars (OPENAI\_API\_KEY, ANTHROPIC\_API\_KEY, GOOGLE\_API\_KEY/GEMINI\_API\_KEY, DEEPSEEK\_API\_KEY, MISTRAL\_API\_KEY, GROQ\_API\_KEY). ## `models` (type: `object`): Optional map of provider name to model id to probe, e.g. {"OpenAI": "gpt-4o-mini"}. Defaults to a cheap reference model per provider. ## `timeoutMs` (type: `integer`): Hard timeout for each provider probe in milliseconds. A timed-out probe is reported with status=timeout and its real elapsed time. ## Actor input object example ```json { "providers": [ "OpenAI", "Anthropic", "Google", "DeepSeek" ], "timeoutMs": 15000 } ``` # Actor output Schema ## `results` (type: `string`): Per-provider latency and token cost records # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("constant_quadruped/llm-latency-cost-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("constant_quadruped/llm-latency-cost-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call constant_quadruped/llm-latency-cost-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=constant_quadruped/llm-latency-cost-monitor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "LLM Latency & Cost Monitor", "description": "Measures cost, speed, and latency across simulated LLM provider APIs for routing optimization.", "version": "1.0", "x-build-id": "o5YSxO5cASMAPX2hP" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/constant_quadruped~llm-latency-cost-monitor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-constant_quadruped-llm-latency-cost-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/constant_quadruped~llm-latency-cost-monitor/runs": { "post": { "operationId": "runs-sync-constant_quadruped-llm-latency-cost-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/constant_quadruped~llm-latency-cost-monitor/run-sync": { "post": { "operationId": "run-sync-constant_quadruped-llm-latency-cost-monitor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "providers": { "title": "Providers list", "type": "array", "description": "LLM providers to probe. Supported: OpenAI, Anthropic, Google, DeepSeek, Mistral, Groq.", "default": [ "OpenAI", "Anthropic", "Google", "DeepSeek" ], "items": { "type": "string" } }, "apiKeys": { "title": "API keys (optional)", "type": "object", "description": "Optional map of provider name to API key, e.g. {\"OpenAI\": \"sk-...\"}. If provided, the actor fires a minimal REAL chat request and measures end-to-end latency. If omitted, it falls back to a real unauthenticated HTTPS latency probe against each provider host. Keys are also read from env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY/GEMINI_API_KEY, DEEPSEEK_API_KEY, MISTRAL_API_KEY, GROQ_API_KEY)." }, "models": { "title": "Model overrides (optional)", "type": "object", "description": "Optional map of provider name to model id to probe, e.g. {\"OpenAI\": \"gpt-4o-mini\"}. Defaults to a cheap reference model per provider." }, "timeoutMs": { "title": "Per-probe timeout (ms)", "type": "integer", "description": "Hard timeout for each provider probe in milliseconds. A timed-out probe is reported with status=timeout and its real elapsed time.", "default": 15000 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```