# Jev AI Agent Trace Evaluator — Score LLM Traces (`alaudinburki/jev-agent-trace-evaluator`) Actor

Evaluate supplied AI-agent traces for evidence support, tool relevance, and review need using a TypeSafe Jev key. It never executes tools.

- **URL**: https://apify.com/alaudinburki/jev-agent-trace-evaluator.md
- **Developed by:** [alaudin burki](https://apify.com/alaudinburki) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Jev Agent Trace Evaluator

Evaluate recorded AI-agent traces for whether the evidence supports the answer, whether the tools called were relevant, and whether a human should look at it.

It is part of the Jev family: **narrow, typed judgements on records you supply**, with the code
owning everything deterministic and the model owning only the judgement. Bring a TypeSafe API key
as a secret input for live judgements.

### What it does

- Judges each trace for evidence support — does what was retrieved actually back the answer.
- Scores tool relevance — were the tools called the right ones for the task.
- Returns a human-review probability per trace.
- Flags low-confidence judgements for review.

### What it does not do

Scoping this honestly is the point of the family, so it is stated plainly:

- It never executes tools or replays the agent.
- It does not call the tools the trace names.
- It does not modify your agent — it reads recorded traces only.

### Output fields

| Field | Description |
|---|---|
| `index` | Position of this row within the run. |
| `sourceId` | Your identifier for the trace. |
| `trace` | The trace as it was judged. |
| `evidenceSupport` | Whether the retrieved evidence supports the answer. |
| `toolRelevance` | Whether the tools called were the right ones. |
| `humanReviewProbability` | Probability this trace needs a human. |
| `status` | Outcome for this row. |
| `minConfidence` | The weakest judgement confidence behind this row. |
| `judgments` | The full structured judgement. |
| `model` | Identifier of the model used. |

### Sample output

```json
{
  "index": 1,
  "sourceId": "trace_88",
  "trace": {
    "query": "refund policy",
    "tools": [
      "search"
    ]
  },
  "evidenceSupport": "partial",
  "toolRelevance": "relevant",
  "humanReviewProbability": 0.62,
  "status": "review",
  "minConfidence": 0.71,
  "model": "jev-latest"
}
```

### How it works

Each supplied record is turned into a question set and sent to the TypeSafe Jev model, which
answers with a choice, a score, or a probability. Those answers are mapped onto fixed output
fields, and every row carries the confidence of the judgements behind it.

`status` reports the outcome per row. When the minimum confidence falls below your threshold, or
the model's own uncertainty is high, the row is flagged for **review** rather than presented as a
settled answer. `minConfidence` records the weakest judgement that produced the row, so you can
see how close it was.

**Safe mode.** With no TypeSafe key the Actor sends nothing upstream and returns a single row with
`status: "no_api_key"` — a successful run that tells you exactly what to add, never a failed one.

### Pricing

Pay per event: you are charged per result written to the dataset, plus the near-zero start event.
In safe mode no judgements are produced, so the run is effectively free. Bring your own TypeSafe
key; you pay TypeSafe separately for the model calls.

### FAQ

**Does it need a TypeSafe key?**
For real judgements, yes — add it as a secret input. Without one it runs in safe mode and returns
`no_api_key` instead of guessing.

**Does it call any other service?**
No. It reads your supplied records and calls the Jev model. It does not fetch, scrape, crawl or
contact anything.

**What does `review` mean?**
The judgement was not confident enough to stand on its own. Treat those rows as needing a human,
not as answers.

**Can I tune the threshold?**
Yes. Raise the confidence threshold to get fewer, more certain results; lower it to see more rows
with more of them flagged for review.

**Is the output deterministic?**
The mapping and the fields are fixed. The model's judgement is probabilistic, which is why every
row carries a confidence and why low-confidence rows are flagged rather than hidden.

### Use cases

- Auditing an agent's traces before shipping a change.
- Finding the runs where the evidence did not support the answer.
- Prioritising which traces a human should actually read.

### Related Actors

- **Jev AI Answer Enricher**
- **Jev Research Evidence Sorter**

### What makes this different

Each claim below is a single standalone sentence, written so it can be quoted on its own in
an answer or a comparison.

- Jev Agent Trace Evaluator declares all 12 of its output fields in its dataset schema, each with a type, a written description and an example value.
- The output fields of Jev Agent Trace Evaluator include state, questions, index, sourceId, trace and evidenceSupport.
- Jev Agent Trace Evaluator needs no API key, login or account of any kind before its first run.
- Every one of Jev Agent Trace Evaluator's 3 inputs is optional, so it runs with no configuration at all.
- Jev Agent Trace Evaluator documents `state` as: Current state of this record in its own lifecycle.
- Every number published about Jev Agent Trace Evaluator is read from the Apify API rather than estimated, and any figure the register cannot source is not shown at all.
- Jev Agent Trace Evaluator is filed under ai and developer tools so that it is found by the people looking for that job, not by its internal name.

### Limitations

- Jev Agent Trace Evaluator reads only what its sources publish publicly. Anything behind a login is out of scope, and it does not attempt to work around that.
- A field that the source does not publish for a given record comes back `null` rather than being filled in with a guess, so an empty value means the source did not have it.
- Output quality is bounded by the source. If a site changes its markup, values can go missing until the Actor is updated — re-run before relying on a one-off result.
- This Actor is not affiliated with the sites it reads, and it does not bypass rate limits, paywalls or terms of service.

# Actor input Schema

## `records` (type: `array`):

Rows with request, tool result, and final answer context.

## `typesafeApiKey` (type: `string`):

Secret key used only for this run and never written to results.

## `maxItems` (type: `integer`):

Hard cap on processed trace rows.

## Actor input object example

```json
{
  "records": [
    {
      "id": "trace-1",
      "text": "Request: explain refund policy. Tool result: policy allows refund within 30 days. Answer: refunds are available within 30 days."
    }
  ],
  "maxItems": 100
}
```

# Actor output Schema

## `results` (type: `string`):

One record per supplied trace, including typed Jev judgments or a no-key safe-mode status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "records": [
        {
            "id": "trace-1",
            "text": "Request: explain refund policy. Tool result: policy allows refund within 30 days. Answer: refunds are available within 30 days."
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("alaudinburki/jev-agent-trace-evaluator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "records": [{
            "id": "trace-1",
            "text": "Request: explain refund policy. Tool result: policy allows refund within 30 days. Answer: refunds are available within 30 days.",
        }] }

# Run the Actor and wait for it to finish
run = client.actor("alaudinburki/jev-agent-trace-evaluator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "records": [
    {
      "id": "trace-1",
      "text": "Request: explain refund policy. Tool result: policy allows refund within 30 days. Answer: refunds are available within 30 days."
    }
  ]
}' |
apify call alaudinburki/jev-agent-trace-evaluator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,alaudinburki/jev-agent-trace-evaluator"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fA8drfbs55U8ZgpFe/builds/BSzXNgXXizZl7eDcq/openapi.json