AI Benchmark Claim Normalizer

Normalize structured AI benchmark claims into evidence-preserving records and explicit, reproducible comparability decisions.

What this does not do

• No scraping arbitrary papers.
• No universal leaderboard.
• No composite quality score.
• No ranking of claims marked non-comparable.
• No LLM extraction in version 1.
• No verification that a reported benchmark number is honest.

Sources and authorization

Version 1 processes only structured JSON supplied directly by the user. A source URL and evidence locator identify where each reported claim came from, but the Actor does not fetch that URL. Use only claims and evidence you are authorized to process. Query strings, fragments, URL credentials, and secret-shaped extension fields are rejected or redacted from output.

Input

Submit 1-1,000 claims. Each needs a claim ID, evidence URL and locator, model name, benchmark name, metric name, and metric value. Comparison-critical context includes benchmark version, split, metric unit/direction, shots, and prompt template. Optional explicit comparison groups produce pairwise matrices without rankings.

{
  "claims": [{
    "claimId": "demo-1",
    "source": {"url": "https://flintglade-tools.github.io/apify-fixtures/benchmarks/demo.html", "locator": "#claim-1"},
    "model": {"name": "ExampleModel", "revision": "abc123"},
    "benchmark": {"name": "ExampleBench", "version": "1.0", "split": "test"},
    "metric": {"name": "accuracy", "value": "0.812", "unit": "fraction", "direction": "higher_is_better"},
    "protocol": {"shots": 0, "promptTemplate": "fixture-v1"}
  }],
  "strictMode": true
}

Output

Every admitted claim row preserves the redacted raw claim, normalized claim, exact alias evidence, context fingerprint, missing fields, source locator, rule versions, warnings, and billing decision. Group rows contain every pairwise state and matching/mismatching field evidence.

{
  "entity": {"key": "demo-1"},
  "status": "ok",
  "charge": {"event": "normalized-claim", "chargeable": true},
  "data": {
    "comparisonState": "comparable",
    "contextFingerprint": "sha256:...",
    "missingCriticalFields": []
  }
}

ALIAS_REGISTRY.json, COMPARABILITY_RULES.json, and ARTIFACT_MANIFEST.json are written to the run key-value store. The run summary is uncharged.

Pricing

The launch event is normalized-claim at $0.02 per admitted claim. A structurally valid normalized claim or explicit insufficient_context/partial refusal is chargeable because the preserved context and refusal evidence are the product. Malformed values, missing source locators, registry failures, charge-limit rows, group summaries, and run summaries are uncharged. For 100 admitted claims, the event charge is $2.00 plus platform usage.

State

This Actor is stateless. It does not maintain baselines or rank claims across runs. The same claim context and registry version produce the same fingerprint and decision; run IDs, dates, reported scores, and observation metadata do not affect the context fingerprint.

Partial failures and negative results

One malformed claim produces an uncharged rejected row without invalidating valid siblings. Strict mode returns insufficient_context when a registry-required field is absent. Non-strict mode returns partially_comparable but still never guesses a value. Exact mismatches produce not_comparable and cite both evidence locators.

Resource limits

• 1-1,000 claims and up to 200 comparison groups.
• Claims JSON is limited to 5 MB and 20 levels of nesting.
• Custom aliases are limited to 500 exact entries.
• Launch memory: 256 MB minimum, 1,024 MB default, 2,048 MB maximum.
• Hosted timeout: 600 seconds.

Data handling and privacy

Claims are written to the default dataset and rule artifacts to the run key-value store under the account's Apify retention settings. The Actor makes no network requests in version 1. Remove run storage in Apify when it is no longer needed. Do not place credentials or private tokens in source URLs or extension fields.

Source, rights, and attribution

The controlled default claim points to a Flintglade-authored public fixture. The Actor records user-provided evidence locators and does not assert ownership, licensing, benchmark validity, or permission to reuse third-party material.

Limitations

Aliases apply only when an exact mapping exists in the selected registry. Percentage/fraction conversion occurs only for a metric whose registry definition explicitly authorizes it. Unknown metrics and unsupported units are preserved and refused, not inferred. Reported values are not fact-checked.

Support

Email support@flintglade.com with the run ID, registry version, and a redacted minimal claim. Never send credentials or private evidence in a support request.

Changelog

See CHANGELOG.md for release history.