CHAPPY Outcome Proof Gate — AI Agent Result Auditor

Does your AI agent say “100 tasks completed” when it can only show 97 receipts?

Outcome Proof Gate audits submitted job-completion claims against item-level evidence records, and returns a deterministic JSON judgment. It helps catch optimistic success reports, duplicated receipts, wrong-run contamination, missing results, and conflicting event states.

Scope matters: this Actor inspects what the caller submits. It does not independently access the destination API, prove that receipt IDs are genuine, or guarantee external-world delivery. An internally consistent result is named SUPPORTED_BY_SUBMITTED_EVIDENCE, never VERIFIED.

Use cases

• A workflow claims it created 100 store listings. Only 97 unique result receipts are present.
• An AI task aggregator accidentally includes another customer or run's result.
• An automation retries and reports the same receipt or item twice.
• Claimed success counts disagree with the status records used to justify them.

Example input

{
  "cases": [
    {
      "jobId": "job-42", "runId": "run-01",
      "claim": {
        "status": "SUCCEEDED",
        "expectedItemIds": ["A", "B", "C"],
        "reportedSuccessCount": 3
      },
      "evidence": [
        {"jobId":"job-42","runId":"run-01","itemId":"A","state":"SUCCEEDED","receiptId":"receipt-A","source":"my-api-readback"},
        {"jobId":"job-42","runId":"run-01","itemId":"B","state":"SUCCEEDED","receiptId":"receipt-B","source":"my-api-readback"}
      ]
    }
  ]
}

Result excerpt

{
  "decision": "EVIDENCE_INCOMPLETE",
  "expected_count": 3,
  "receipt_supported_count": 2,
  "missing_count": 1,
  "missing_item_ids_sample": ["C"],
  "external_verification": false,
  "evidence_trust": "SUBMITTED_ONLY",
  "claim_consistent_with_evidence": false
}

Decisions

Decision	Meaning
SUPPORTED_BY_SUBMITTED_EVIDENCE	All declared items have one successful submitted receipt, with consistent job/run/count. Not externally verified
EVIDENCE_INCOMPLETE	Expected item receipts missing or a failure/pending status appeared
IDENTITY_MISMATCH	Evidence references a different job or run
EVIDENCE_CONFLICT	Duplicate items/receipt IDs or unexpected items
CLAIM_CONFLICT	Claimed count/status differs from the submitted evidence
INVALID_INPUT	Required fields or limits are invalid

Limits, privacy, and trust

1–50 cases per run; 1–500 expected IDs and up to 1,000 evidence records per case. Use pseudonymous job and item IDs. Do not send secrets, payment details, personal data, raw credentials, or private customer materials. Data may be retained according to Apify storage settings.

The source field is a caller-supplied label, not authentication. Even “independent-verifier” is untrusted unless separately confirmed by your integration. Future versions may optionally fetch actual destination state with a customer-controlled credential boundary; this version never does so.

For AI agents

Call this Actor before announcing completion. If decision is SUPPORTED_BY_SUBMITTED_EVIDENCE, follow up with genuine destination readback before claiming real-world success. If the decision is any other value, use the flags and missing IDs to investigate or retry safely.

Pricing

Initial public price: $0.003 USD per audited, syntactically valid job case, billed through the custom audit-case event (pay per event). Inputs that fail validation are returned with INVALID_INPUT without an audit-case charge. The buyer's max-run spending limit is respected; if exhausted, the Actor stops processing additional cases. For this PPE mode, platform usage is paid from the developer payout rather than charged to the buyer. The price is an initial experiment and does not guarantee verified external-world outcomes.

Developed by CHAPPY. Not affiliated with Apify.

Critical distinction: execution vs business outcome

The business goal must define which source-result statuses mean success before generating receipt state values. For example, a postcode lookup run can finish successfully while an individual lookup returns NOT_FOUND; if the goal is a valid address match, that record must be supplied as FAILED (or otherwise not SUCCEEDED). Treating every processed row as success will produce a misleading SUPPORTED_BY_SUBMITTED_EVIDENCE result. This tool checks submitted receipt consistency, not whether your success criterion was correctly chosen. Use independent API readback when the actual goal matters.