CI Release Evidence Gate

What does CI Release Evidence Gate do?

Compare the CI evidence you have with the evidence your release requires. Supply a small manifest of required job, test-shard and artifact identities, then attach normalized exports for one intended run and commit. The Actor returns a review queue for missing shards, skipped required jobs, wrong commits, failed or empty test shards, duplicate artifact records and inconsistent SHA-256 metadata.

Use the same manifest for each release or after changing a CI matrix. Each finding names a rule, the exact expected identity, the observed evidence IDs and JSON Pointers into your submitted input. A job reported as successful cannot hide an absent required shard. Duplicate observations remain visible; the Actor does not choose a convenient passing record.

When is this useful?

Release engineers can check that all declared matrix shards are represented before reviewing a release. Platform teams can compare required build artifacts with their recorded commit, run and digest. Teams combining several report sources can download one JSON, CSV and standalone HTML evidence report through the Apify API or a saved task. A recurring schedule requires a caller or integration to supply updated evidence; resubmitting yesterday's input does not collect today's results.

This checks declared evidence completeness and consistency. It does not execute tests, fetch repositories, read branch protection, evaluate workflow conditions or approve a deployment. meets-declared-rules means only that the submitted required identities satisfy the submitted policy. It is not proof of code correctness, artifact authenticity or complete test coverage.

How do I prepare CI evidence?

1. Choose one exact run attempt and commit. Define stable literal job/shard/artifact names in the manifest. IDs such as 001 remain strings, distinct from 1; case and spaces are significant.
2. Export job metadata with tooling you already own. Keep repository credentials on your computer or in your existing CI environment. Upload metadata and test counts, not tokens, logs, source code or customer payloads.
3. Emit a normalized count record from each test shard and collect artifact metadata. Use the export recipe below. Missing facts can be omitted or null; they become unknown findings rather than invented success.
4. Declare coverage.jobs, coverage.shards and coverage.artifacts as complete, incomplete or unknown. Only call an export complete after checking pagination and the intended attempt/scope. An absent identity in an incomplete export is unknown, not a verified failure.
5. Run the Actor. Review findings and source references; download the HTML with the attachment link and open it locally.

GitHub Actions export recipe

The GitHub workflow jobs API exposes identity, run, commit, status, conclusion and timestamps. With your existing authenticated GitHub CLI, export the specific attempt, including all pages:

gh api --paginate --slurp \
  'repos/OWNER/REPO/actions/runs/RUN_ID/attempts/ATTEMPT/jobs?per_page=100' \
  > jobs.pages.json

This is a command you execute in your environment; the Actor never receives or uses a GitHub token. Normalize the saved pages locally, retaining every record:

import { readFileSync, writeFileSync } from 'node:fs';
const attempt = '1'; // same specific attempt as the export URL
const pages = JSON.parse(readFileSync('jobs.pages.json', 'utf8'));
const jobs = pages.flatMap(p => p.jobs).map(j => ({
  evidenceId: String(j.id), identity: j.name,
  runId: `${j.run_id}/${attempt}`, commit: j.head_sha,
  status: j.status, conclusion: j.conclusion,
  timestamp: j.completed_at || j.started_at
    ? new Date(j.completed_at || j.started_at).toISOString() : null,
}));
writeFileSync('jobs.normalized.json', JSON.stringify(jobs, null, 2));

Set top-level runId to that same RUN_ID/ATTEMPT, and commit to the intended full commit string. Job identity must match the API's literal name, including matrix suffixes. Do not silently deduplicate names: if names collide, rename jobs or define an explicit stable alias during your normalization and use it consistently in the manifest. This local recipe does not mark pagination or coverage complete for you.

For each test shard, have your existing reporter or wrapper export counts with the same release provenance. For example, a shard produced by job linux-001 can write:

{"evidenceId":"shard-01","identity":"unit-01","jobIdentity":"linux-001","runId":"run-001","commit":"commit-001","timestamp":"2026-10-01T11:46:00Z","total":10,"passed":9,"failed":0,"skipped":1}

Get those counts from the report generated by your test runner. Do not estimate them from log lines. total must equal passed + failed + skipped, and all four counts must be nonnegative safe integers. The minimum uses passed + failed, excluding skipped tests; any failed count still creates a finding. This first version consumes normalized JSON rather than parsing JUnit XML.

GitLab export recipe

Use your existing authorized GitLab tooling to export all pages of jobs for the intended pipeline, retaining retry records. For example, glab api --paginate 'projects/PROJECT_ID/pipelines/PIPELINE_ID/jobs?include_retried=true&per_page=100' exports job arrays; combine its JSON arrays locally with jq -s add before normalization. Map id to string evidenceId, name to identity, pipeline.id to string runId, commit.id to commit, and finished_at (or started_at) to a strict UTC timestamp. Map success to completed/success, failed to completed/failure, canceled to completed/cancelled, skipped to completed/skipped, running to in_progress/null, and created/pending to queued/null. Map other states to completed/unknown for review; do not label them success. Review repeated names from retries and matrix jobs and define the intended scope explicitly. Retained duplicates become findings.

GitLab test report documentation explains the distinction between displayed reports and job success. Emit shard-count JSON from your existing reporter just as above. Export artifact identities and generation timestamps from your build metadata; if checking a digest, compute SHA-256 locally and submit both the trusted expected digest and recorded observed digest. The Actor compares those strings and does not download or hash artifacts.

Input limits and policies

The Input tab provides a worked synthetic release with a successful build/shard, a missing integration shard, a job from the wrong commit, a skipped required job and duplicate conflicting artifact evidence. The source bundle also includes examples/complete.json for a three-identity example that meets the declared rules.

One run accepts at most 2 MB JSON, 500 combined required identities and 5,000 combined observed records. All three expected and observed arrays are required; use empty arrays for unrequested kinds. Required identities cannot repeat within a kind. Every expected shard must reference an expected job. Unknown input fields, malformed SHA-256 strings, invalid count arithmetic and ambiguous timestamps are rejected before a report event is charged.

asOf is mandatory UTC in YYYY-MM-DDTHH:mm:ssZ or YYYY-MM-DDTHH:mm:ss.sssZ. Optional maxEvidenceAgeMinutes is a positive integer up to 525,600; equality is accepted. Stale evidence gets a warning requiring review, and future evidence gets an error. Missing run, commit, conclusion, timestamp or required digest gets an explicit unknown finding. A shard cannot satisfy requirements while its associated job has findings or unknown evidence. Jobs accept success by default; you may explicitly accept neutral. Skipped, cancelled, failed and pending jobs cannot satisfy a required job.

Output and cost

The price is $0.15 per completed report, with platform usage included. A report with findings is still a completed report. Validation failure or insufficient budget produces no completed-report charge. A new run is billable again.

Download	Contents
Full report dataset / OUTPUT JSON	One complete report with summary, declared scope, rules, evidence rows and findings
evidence.csv	One row per required or unexpected identity, including all observation IDs and shard counts
findings.csv	Flat review queue with rule, severity, expected/observed values and source references
report.html	Escaped standalone readable report with no external scripts or resources

CSV neutralizes spreadsheet formula prefixes; the complete JSON preserves original strings. All export sizes are checked before billing: full JSON must be below 8 MB and each export below 9 MB. Heavily conflicting evidence may expand beyond those limits; split the report if needed.

Exports are saved before publishing the single report dataset item and its event. An interruption may leave partial, uncharged convenience files. Resuming the same run regenerates them and verifies the saved report's input fingerprint and full deterministic contents. A persisted valid report and charge are reused without a second dataset item or event. If a charged dataset was deleted or altered, recovery refuses to guess; inspect or restore it. These protections apply to same-run recovery, not separate new runs.

Questions and support

Use the Issues tab for unexpected normalization cases. Include a small sanitized fixture and the relevant source pointers. Upload only metadata you are authorized to share with Apify. No external API, repository login or model subscription is needed by this Actor. For programmatic use, the API tab exposes run and result-download endpoints. Human release policy remains responsible for deciding what evidence is required and what a finding means for a deployment.