# Dataset Cost Estimator — LLM Embedding Budget (`fg-ei/dataset-llm-cost-estimator`) Actor

Estimate USD token costs for an Apify dataset before RAG upsert. Compare chat & embedding models; optional fail-if-over-budget. No LLM API keys.

- **URL**: https://apify.com/fg-ei/dataset-llm-cost-estimator.md
- **Developed by:** [FG EI](https://apify.com/fg-ei) (community)
- **Categories:** AI, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.03 / cost report

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset LLM Cost Estimator — Tokens & Embedding Budget

Estimate how much it will cost to **embed** or **run chat models over an Apify dataset** *before* you spend money with an LLM provider. Point the Actor at a dataset (for example the output of Website Content Crawler) or at inline JSON rows. You get a deterministic token and USD report for several chat and embedding models, plus an optional **fail-if-over-budget gate** for agents and CI pipelines.

- No LLM API key needed: the Actor never calls OpenAI, Anthropic, Google or any other provider.
- Deterministic: the same input always gives the same numbers.
- Output: dataset rows, a full JSON report (`OUTPUT`) and a Markdown report (`REPORT`).

> Token counts are a **character-based heuristic**, not an exact tokenizer. Use them for budgeting and model comparison, not as an invoice. See [Limitations](#limitations).

### What you get

- Token and USD estimates for **embedding** models (e.g. `text-embedding-3-small` vs `text-embedding-3-large`) and **chat** models (e.g. `gpt-4o-mini`, `gpt-4o`, `claude-haiku-4.5`, `gemini-2.0-flash`)
- Token statistics per row (p50 / p90 / p99 / max) to size chunks and context windows
- A budget gate: `gateStatus: PASS | FAIL` in the summary; with `failIfOverBudgetUsd` enabled the run itself fails with `BUDGET_GATE_FAIL` after the reports are written
- The price table version used (`priceTableVersion`) in every report

This Actor is **not** a scraper, **not** an LLM processor and **not** an exact BPE tokenizer.

### Use cases

1. **After a crawl, before a RAG upsert**: run Website Content Crawler or RAG Web Browser, pass the resulting dataset ID, compare embedding costs before writing to your vector database.
2. **Budget gate for agents / CI**: set `budgetUsd`, `budgetBasis` and `failIfOverBudgetUsd: true`. The run fails when the estimate exceeds your budget, so the next step of your pipeline does not start.
3. **Compare chat vs embedding costs on the same corpus** in one run.
4. **Quick test with inline rows**: paste JSON objects in `items` without creating a dataset.

### Quick start

1. Open the **Input** tab. The Inline JSON items field is prefilled with a 3-row sample: click **Start** to see a full report.
2. To analyze your own crawl: **clear Inline JSON items**, paste your dataset ID in **Apify Dataset ID** (keep *Allow remote dataset fetch* enabled) and start.
3. Optionally set **Budget (USD)** and **Budget comparison basis**, and enable **Fail Actor run if over budget** for automation.
4. Read the **Output** tab: cost estimate rows, Markdown report and full JSON report.

Example input (API):

```json
{
  "reportName": "RAG crawl sample budget",
  "items": [
    {"url": "https://example.com/a", "text": "Deterministic token budgeting helps agent pipelines avoid surprise embedding bills before vector upsert."},
    {"url": "https://example.com/b", "markdown": "Second document with enough characters to exercise percentile stats and multi-model cost rows."},
    {"url": "https://example.com/empty"}
  ],
  "textFields": ["text", "markdown"],
  "chatModels": ["gpt-4o-mini", "gpt-4o"],
  "embeddingModels": ["text-embedding-3-small", "text-embedding-3-large"],
  "assumedOutputTokensPerItem": 50,
  "budgetUsd": 1.0,
  "budgetBasis": "cheapest_embedding",
  "failIfOverBudgetUsd": false
}
```

Analyze an existing dataset instead:

```json
{
  "datasetId": "<your-dataset-id>",
  "allowRemoteFetch": true,
  "maxItems": 1000,
  "budgetUsd": 5,
  "budgetBasis": "most_expensive_embedding",
  "failIfOverBudgetUsd": true
}
```

### Input

Provide **exactly one** source: `items` or `datasetId`.

| Field | Description | Default |
|---|---|---|
| `reportName` | Label shown in the report | `Dataset LLM cost estimate` |
| `items` | Inline JSON rows (max 5,000) | – |
| `datasetId` | ID or name of an Apify dataset you own (read-only access requested) | – |
| `allowRemoteFetch` | Must be `true` to read `datasetId` | `true` |
| `textFields` | Fields concatenated as billable text | `text, markdown, content, body, pageContent, description` |
| `chatModels` | Chat model IDs to price | `gpt-4o-mini, gpt-4o, claude-haiku-4.5, gemini-2.0-flash` |
| `embeddingModels` | Embedding model IDs to price | `text-embedding-3-small, text-embedding-3-large` |
| `maxItems` | Safety cap on analyzed rows (1–50,000) | `1000` |
| `assumedOutputTokensPerItem` | Assumed completion tokens per non-empty row for chat estimates | `0` |
| `budgetUsd` | Optional USD budget for the gate | – |
| `budgetBasis` | `cheapest_embedding`, `most_expensive_embedding`, `cheapest_chat_input`, `most_expensive_chat_input` | `cheapest_embedding` |
| `failIfOverBudgetUsd` | Fail the run (after writing reports) if over budget; requires `budgetUsd` | `false` |

**Supported models (price table `2026-09-11`)**

- Chat: `gpt-4o-mini`, `gpt-4o`, `gpt-4.1-mini`, `claude-haiku-4.5`, `claude-sonnet-4.6`, `gemini-2.0-flash`, `llama-3.3-70b`
- Embedding: `text-embedding-3-small`, `text-embedding-3-large`, `voyage-3-lite`, `voyage-3`

Unknown model IDs are rejected with an explicit `INPUT_ERROR`.

**Analyzing the output of another run:** open that run in Apify Console, copy its default dataset ID and pass it in `datasetId`. This Actor uses Apify **limited permissions** and requests read access to that dataset only. A `runId` input is **not supported** in this version: an input containing a non-empty `runId` is rejected with `INPUT_ERROR` (only the Actor start event is charged).

### Output

**Dataset** (Output tab, views *Overview* and *Model estimates*):

- 1 row `type: cost_estimate_summary`: `gateStatus`, `itemsAnalyzed`, `itemsWithText`, `totalChars`, `totalTokensRefCl100kApprox`, `tokenPercentilesRef`, `budgetUsd`, `budgetBasis`, `budgetBasisModel`, `budgetBasisCostUsd`, `overBudget`, `priceTableVersion`, `disclaimer`…
- 1 row per embedding model `type: embedding_model_estimate`: `model`, `inputTokens`, `inputCostUsd`, `totalCostUsd`, `usdPer1MInput`
- 1 row per chat model `type: chat_model_estimate`: `model`, `inputTokens`, `assumedOutputTokens`, `inputCostUsd`, `outputCostUsd`, `totalCostUsd`, `usdPer1MInput`, `usdPer1MOutput`

**Key-value store**: `OUTPUT` (full JSON report, including a `ppe` ledger of the events charged by the run) and `REPORT` (Markdown).

Sample summary (from the example input above):

```json
{
  "type": "cost_estimate_summary",
  "reportName": "RAG crawl sample budget",
  "gateStatus": "PASS",
  "itemsAnalyzed": 3,
  "itemsWithText": 2,
  "totalTokensRefCl100kApprox": 50,
  "budgetUsd": 1.0,
  "budgetBasis": "cheapest_embedding",
  "budgetBasisModel": "text-embedding-3-small",
  "budgetBasisCostUsd": 1e-06,
  "overBudget": false
}
```

Sample Markdown report:

```markdown
## RAG crawl sample budget

- Gate: **PASS**
- Items analyzed: 3 / available 3
- Items with text: 2 · empty: 1

### Embedding estimates (cheapest first)

| Model | Tokens | USD |
|---|---:|---:|
| text-embedding-3-small | 50 | 0.000001 |
| text-embedding-3-large | 50 | 0.000006 |

### Chat estimates (cheapest total first)

| Model | Input tokens | Assumed output tokens | Input USD | Output USD | Total USD |
|---|---:|---:|---:|---:|---:|
| gpt-4o-mini | 50 | 100 | 0.000008 | 0.000060 | 0.000068 |
| gpt-4o | 50 | 100 | 0.000125 | 0.001000 | 0.001125 |
```

Errors are explicit in the run log: `INPUT_ERROR` (invalid input), `PROCESS_ERROR` (rows cannot be processed, e.g. a text cap is exceeded), `UNHANDLED_ERROR` (unexpected failure, e.g. the dataset cannot be read), `BUDGET_GATE_FAIL` (over budget with `failIfOverBudgetUsd`).

### How it works

1. Reads rows from `items` or `datasetId` (up to `maxItems`)
2. Extracts text from the configured `textFields`
3. Estimates tokens with a versioned characters-per-token heuristic (about 4 chars/token, 3.8 for Claude models)
4. Applies the static USD price table (per 1M tokens)
5. Writes the dataset rows, `OUTPUT` and `REPORT`, then applies the optional budget gate

### Pricing

This Actor uses **pay per event** pricing. Events as configured:

| Event | Price | When it is charged |
|---|---:|---|
| Actor start (`apify-actor-start`) | $0.00005 | Once when the run starts (one event per GB of memory; memory is capped at 1 GB) |
| Rows analyzed (`rows-analyzed`) | $0.00005 per row | Once per analyzed row (`itemsAnalyzed`, i.e. rows admitted after `maxItems`), only after the report is written |
| Cost report (`cost-report`) | $0.03 | Once per run, only after `OUTPUT` / `REPORT` are written |

**Typical cost per run:** `$0.00005 + $0.00005 × itemsAnalyzed + $0.03`

| Rows analyzed | Event cost |
|---:|---:|
| 3 (example input) | ≈ $0.0302 |
| 100 | ≈ $0.035 |
| 1,000 | ≈ $0.08 |
| 10,000 | ≈ $0.53 |
| 50,000 (maximum `maxItems`) | ≈ $2.53 |

- Invalid input or a failure before the report is written: only the Actor start event is charged.
- Budget gate FAIL: the reports are written, so rows analyzed and the cost report are charged.
- You control the maximum with `maxItems` and with Apify's maximum cost per run setting.
- No LLM provider cost: the Actor does not call any LLM API. The pricing panel on this page is the authoritative source for current prices.

### Limitations

- Token counts are a **heuristic** (characters per token), not tiktoken / exact BPE. Calibrate against your provider's tokenizer for invoices.
- Model prices come from a **static table** dated `2026-09-11`; provider prices may change. The table version is written in every report.
- No chunking, embedding, scraping or LLM calls are performed.
- Caps: `maxItems` ≤ 50,000; inline `items` ≤ 5,000; 500,000 characters per row; 20,000,000 characters per run.
- Only one source per run (`items` or `datasetId`); `runId` is not supported.

### Typical pipeline

```
Website Content Crawler / RAG Web Browser  →  Dataset LLM Cost Estimator  →  (if PASS) your embedding / LLM pipeline
```

### Permissions and data

- Runs with Apify **limited permissions**: it reads its own input, the dataset you pass, and writes to its own default storages.
- Text is processed inside the Actor container and is never sent to an LLM provider.
- Results are kept in your run storages according to your Apify retention settings. You remain responsible for the lawfulness of the content you analyze (e.g. GDPR).

### FAQ

**Why not a local script?** Your crawl already lives in an Apify dataset, and agents, webhooks and schedules can call an Actor directly. The budget gate is the run status itself, so an orchestrator can stop the pipeline without custom code.

**How accurate are the token counts?** Good enough for budget gates and relative comparisons between models; not an exact invoice. Every report includes a disclaimer.

**Do I need an OpenAI / Anthropic key?** No.

**Support:** open an issue on the Actor's Issues tab. Never paste secrets or API keys in issues.

# Actor input Schema

## `reportName` (type: `string`):

Friendly label shown in the Markdown REPORT and in the summary row (reportName).

## `items` (type: `array`):

Paste rows here for quick tests (max 5,000 rows). Each object should contain text fields such as text or markdown. Mutually exclusive with Apify Dataset ID: clear this field before using a dataset.

## `datasetId` (type: `string`):

ID (or name) of an existing Apify dataset you own, for example the default dataset of a Website Content Crawler run (open the run and copy its dataset ID). Read-only access is requested for this dataset only. Clear Inline JSON items when using this field.

## `allowRemoteFetch` (type: `boolean`):

Must be enabled to read a dataset via Apify Dataset ID. Ignored for Inline JSON items.

## `textFields` (type: `array`):

Field names whose values are concatenated as billable text for each row. Defaults cover common crawler outputs: text, markdown, content, body, pageContent, description.

## `chatModels` (type: `array`):

Model IDs to price. Empty = defaults (gpt-4o-mini, gpt-4o, claude-haiku-4.5, gemini-2.0-flash). Supported: gpt-4o-mini, gpt-4o, gpt-4.1-mini, claude-haiku-4.5, claude-sonnet-4.6, gemini-2.0-flash, llama-3.3-70b. Unknown IDs fail validation.

## `embeddingModels` (type: `array`):

Embedding model IDs to price. Empty = defaults (text-embedding-3-small, text-embedding-3-large). Supported: text-embedding-3-small, text-embedding-3-large, voyage-3-lite, voyage-3. Unknown IDs fail validation.

## `maxItems` (type: `integer`):

Hard cap on rows analyzed. Rows beyond the cap are ignored. The rows-analyzed event is charged per analyzed row, so this also caps that charge (1,000 rows = $0.05).

## `assumedOutputTokensPerItem` (type: `integer`):

Assumed completion tokens per non-empty row when estimating chat cost. Use 0 to price input tokens only.

## `budgetUsd` (type: `number`):

Optional USD ceiling compared to the estimate selected in 'Budget comparison basis'. Leave empty for a report without gate.

## `budgetBasis` (type: `string`):

Which estimate is compared to Budget (USD).

## `failIfOverBudgetUsd` (type: `boolean`):

When enabled (requires Budget), the Actor writes OUTPUT and REPORT, then fails the run with BUDGET_GATE_FAIL if the estimate exceeds the budget. Use as a CI / agent gate.

## Actor input object example

```json
{
  "reportName": "Dataset LLM cost estimate",
  "items": [
    {
      "url": "https://example.com/a",
      "title": "Page A",
      "text": "Deterministic token budgeting helps agent pipelines avoid surprise embedding bills before vector upsert."
    },
    {
      "url": "https://example.com/b",
      "title": "Page B",
      "markdown": "Second document with enough characters to exercise percentile stats and multi-model cost rows."
    },
    {
      "url": "https://example.com/empty",
      "title": "Empty"
    }
  ],
  "allowRemoteFetch": true,
  "textFields": [
    "text",
    "markdown",
    "content",
    "body",
    "pageContent",
    "description"
  ],
  "maxItems": 1000,
  "assumedOutputTokensPerItem": 0,
  "budgetBasis": "cheapest_embedding",
  "failIfOverBudgetUsd": false
}
```

# Actor output Schema

## `overview` (type: `string`):

Summary row (gateStatus, itemsAnalyzed, budget) followed by one row per priced model.

## `report` (type: `string`):

Human-readable Markdown REPORT: gate, token stats, embedding and chat cost tables.

## `output` (type: `string`):

OUTPUT record: summary, embeddingEstimates, chatEstimates, shouldFailRun and ppe charge ledger.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "url": "https://example.com/a",
            "title": "Page A",
            "text": "Deterministic token budgeting helps agent pipelines avoid surprise embedding bills before vector upsert."
        },
        {
            "url": "https://example.com/b",
            "title": "Page B",
            "markdown": "Second document with enough characters to exercise percentile stats and multi-model cost rows."
        },
        {
            "url": "https://example.com/empty",
            "title": "Empty"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fg-ei/dataset-llm-cost-estimator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "items": [
        {
            "url": "https://example.com/a",
            "title": "Page A",
            "text": "Deterministic token budgeting helps agent pipelines avoid surprise embedding bills before vector upsert.",
        },
        {
            "url": "https://example.com/b",
            "title": "Page B",
            "markdown": "Second document with enough characters to exercise percentile stats and multi-model cost rows.",
        },
        {
            "url": "https://example.com/empty",
            "title": "Empty",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("fg-ei/dataset-llm-cost-estimator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    {
      "url": "https://example.com/a",
      "title": "Page A",
      "text": "Deterministic token budgeting helps agent pipelines avoid surprise embedding bills before vector upsert."
    },
    {
      "url": "https://example.com/b",
      "title": "Page B",
      "markdown": "Second document with enough characters to exercise percentile stats and multi-model cost rows."
    },
    {
      "url": "https://example.com/empty",
      "title": "Empty"
    }
  ]
}' |
apify call fg-ei/dataset-llm-cost-estimator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fg-ei/dataset-llm-cost-estimator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/onWJsfI9YNISnihJI/builds/zzx9FjV23xacXwinH/openapi.json