Jev Agent Trace Evaluator

Evaluate recorded AI-agent traces for whether the evidence supports the answer, whether the tools called were relevant, and whether a human should look at it.

It is part of the Jev family: narrow, typed judgements on records you supply, with the code owning everything deterministic and the model owning only the judgement. Bring a TypeSafe API key as a secret input for live judgements.

What it does

• Judges each trace for evidence support — does what was retrieved actually back the answer.
• Scores tool relevance — were the tools called the right ones for the task.
• Returns a human-review probability per trace.
• Flags low-confidence judgements for review.

What it does not do

Scoping this honestly is the point of the family, so it is stated plainly:

• It never executes tools or replays the agent.
• It does not call the tools the trace names.
• It does not modify your agent — it reads recorded traces only.

Output fields

Field	Description
index	Position of this row within the run.
sourceId	Your identifier for the trace.
trace	The trace as it was judged.
evidenceSupport	Whether the retrieved evidence supports the answer.
toolRelevance	Whether the tools called were the right ones.
humanReviewProbability	Probability this trace needs a human.
status	Outcome for this row.
minConfidence	The weakest judgement confidence behind this row.
judgments	The full structured judgement.
model	Identifier of the model used.

Sample output

{
  "index": 1,
  "sourceId": "trace_88",
  "trace": {
    "query": "refund policy",
    "tools": [
      "search"
    ]
  },
  "evidenceSupport": "partial",
  "toolRelevance": "relevant",
  "humanReviewProbability": 0.62,
  "status": "review",
  "minConfidence": 0.71,
  "model": "jev-latest"
}

How it works

Each supplied record is turned into a question set and sent to the TypeSafe Jev model, which answers with a choice, a score, or a probability. Those answers are mapped onto fixed output fields, and every row carries the confidence of the judgements behind it.

status reports the outcome per row. When the minimum confidence falls below your threshold, or the model's own uncertainty is high, the row is flagged for review rather than presented as a settled answer. minConfidence records the weakest judgement that produced the row, so you can see how close it was.

Safe mode. With no TypeSafe key the Actor sends nothing upstream and returns a single row with status: "no_api_key" — a successful run that tells you exactly what to add, never a failed one.

Pricing

Pay per event: you are charged per result written to the dataset, plus the near-zero start event. In safe mode no judgements are produced, so the run is effectively free. Bring your own TypeSafe key; you pay TypeSafe separately for the model calls.

FAQ

Does it need a TypeSafe key? For real judgements, yes — add it as a secret input. Without one it runs in safe mode and returns no_api_key instead of guessing.

Does it call any other service? No. It reads your supplied records and calls the Jev model. It does not fetch, scrape, crawl or contact anything.

What does review mean? The judgement was not confident enough to stand on its own. Treat those rows as needing a human, not as answers.

Can I tune the threshold? Yes. Raise the confidence threshold to get fewer, more certain results; lower it to see more rows with more of them flagged for review.

Is the output deterministic? The mapping and the fields are fixed. The model's judgement is probabilistic, which is why every row carries a confidence and why low-confidence rows are flagged rather than hidden.

Use cases

• Auditing an agent's traces before shipping a change.
• Finding the runs where the evidence did not support the answer.
• Prioritising which traces a human should actually read.

Related Actors

• Jev AI Answer Enricher
• Jev Research Evidence Sorter

What makes this different

Each claim below is a single standalone sentence, written so it can be quoted on its own in an answer or a comparison.

• Jev Agent Trace Evaluator declares all 12 of its output fields in its dataset schema, each with a type, a written description and an example value.
• The output fields of Jev Agent Trace Evaluator include state, questions, index, sourceId, trace and evidenceSupport.
• Jev Agent Trace Evaluator needs no API key, login or account of any kind before its first run.
• Every one of Jev Agent Trace Evaluator's 3 inputs is optional, so it runs with no configuration at all.
• Jev Agent Trace Evaluator documents state as: Current state of this record in its own lifecycle.
• Every number published about Jev Agent Trace Evaluator is read from the Apify API rather than estimated, and any figure the register cannot source is not shown at all.
• Jev Agent Trace Evaluator is filed under ai and developer tools so that it is found by the people looking for that job, not by its internal name.

Limitations

• Jev Agent Trace Evaluator reads only what its sources publish publicly. Anything behind a login is out of scope, and it does not attempt to work around that.
• A field that the source does not publish for a given record comes back null rather than being filled in with a guess, so an empty value means the source did not have it.
• Output quality is bounded by the source. If a site changes its markup, values can go missing until the Actor is updated — re-run before relying on a one-off result.
• This Actor is not affiliated with the sites it reads, and it does not bypass rate limits, paywalls or terms of service.