✅ Agent Claim Checker — catch AI agents saying "tests pass" when they don't

AI coding agents finish with confident reports: "Fixed the bug. All tests pass. 2 files changed." Sometimes the tests never ran, the last run failed, or the agent edited the tests themselves.

Agent Claim Checker compares the report with what the tools really returned and tells you which claims are not backed by evidence.

What it checks

Claim in the report	Flagged when
"all tests pass" / 「テストはすべて通りました」	no test run in the log, the last test run failed, or test files were edited in the same session
"fixed" / 「修正しました」	there is no failing test run followed by a passing one
"N files changed"	N does not match the files actually edited
"done" / 「完了」	nothing ran successfully after the last change

Works with pytest, npm/yarn/pnpm test, go test, cargo test, jest, vitest, rspec, phpunit, mvn/gradle test and more.

Input

{
  "report": "Fixed the login bug. All tests pass. 2 files changed.",
  "evidence": [
    {"tool": "Edit", "path": "src/login.py"},
    {"tool": "Bash", "command": "pytest -q", "exit_code": 1}
  ]
}

Or send many reports at once in checks.

Output example

{
  "verdict": "unsupported",
  "claims": ["file_count", "fixed", "tests_pass"],
  "problems": [
    {"claim": "tests_pass", "why": "says tests pass, but the last test run did not succeed (exit code 1)"},
    {"claim": "fixed", "why": "says fixed, but the evidence has no failing test run followed by a passing one"},
    {"claim": "file_count", "why": "says 2 files changed, but the evidence shows 1"}
  ],
  "testRuns": 1,
  "filesChanged": 1
}

supported → the claims match the log · unsupported → ask the agent to prove it · no_claims → nothing to check.

Pricing

Pay per checked report (see the Pricing tab). Apart from Apify's standard start fee there is no per-run charge.

Notes

• Rule-based and fast: it only knows the claim phrases and test commands listed above, so unusual wording can slip through. It checks consistency with the log you give it; it cannot see work that is not in the log.
• Your text is only read inside your run and not sent anywhere else.

Part of AIzilla CAGE — guardrail systems for AI coding agents that run on your own machine: https://aizilla.neuro-hive.jp/en/

Made by KOTARO Data Works 🇯🇵