# AI Benchmark Claim Normalizer (`flintglade/ai-benchmark-claim-normalizer`) Actor Normalize structured AI benchmark claims and produce evidence-backed comparability refusals when versions, splits, metrics, prompts, shots, or model settings do not align. - **URL**: https://apify.com/flintglade/ai-benchmark-claim-normalizer.md - **Developed by:** [Flintglade](https://apify.com/flintglade) (community) - **Categories:** AI, Developer tools, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $20.00 / 1,000 normalized claims This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AI Benchmark Claim Normalizer Normalize structured AI benchmark claims into evidence-preserving records and explicit, reproducible comparability decisions. ### What this does not do - No scraping arbitrary papers. - No universal leaderboard. - No composite quality score. - No ranking of claims marked non-comparable. - No LLM extraction in version 1. - No verification that a reported benchmark number is honest. ### Sources and authorization Version 1 processes only structured JSON supplied directly by the user. A source URL and evidence locator identify where each reported claim came from, but the Actor does not fetch that URL. Use only claims and evidence you are authorized to process. Query strings, fragments, URL credentials, and secret-shaped extension fields are rejected or redacted from output. ### Input Submit 1-1,000 claims. Each needs a claim ID, evidence URL and locator, model name, benchmark name, metric name, and metric value. Comparison-critical context includes benchmark version, split, metric unit/direction, shots, and prompt template. Optional explicit comparison groups produce pairwise matrices without rankings. ```json { "claims": [{ "claimId": "demo-1", "source": {"url": "https://flintglade-tools.github.io/apify-fixtures/benchmarks/demo.html", "locator": "#claim-1"}, "model": {"name": "ExampleModel", "revision": "abc123"}, "benchmark": {"name": "ExampleBench", "version": "1.0", "split": "test"}, "metric": {"name": "accuracy", "value": "0.812", "unit": "fraction", "direction": "higher_is_better"}, "protocol": {"shots": 0, "promptTemplate": "fixture-v1"} }], "strictMode": true } ```` ### Output Every admitted claim row preserves the redacted raw claim, normalized claim, exact alias evidence, context fingerprint, missing fields, source locator, rule versions, warnings, and billing decision. Group rows contain every pairwise state and matching/mismatching field evidence. ```json { "entity": {"key": "demo-1"}, "status": "ok", "charge": {"event": "normalized-claim", "chargeable": true}, "data": { "comparisonState": "comparable", "contextFingerprint": "sha256:...", "missingCriticalFields": [] } } ``` `ALIAS_REGISTRY.json`, `COMPARABILITY_RULES.json`, and `ARTIFACT_MANIFEST.json` are written to the run key-value store. The run summary is uncharged. ### Pricing The launch event is `normalized-claim` at $0.02 per admitted claim. A structurally valid normalized claim or explicit `insufficient_context`/partial refusal is chargeable because the preserved context and refusal evidence are the product. Malformed values, missing source locators, registry failures, charge-limit rows, group summaries, and run summaries are uncharged. For 100 admitted claims, the event charge is $2.00 plus platform usage. ### State This Actor is stateless. It does not maintain baselines or rank claims across runs. The same claim context and registry version produce the same fingerprint and decision; run IDs, dates, reported scores, and observation metadata do not affect the context fingerprint. ### Partial failures and negative results One malformed claim produces an uncharged rejected row without invalidating valid siblings. Strict mode returns `insufficient_context` when a registry-required field is absent. Non-strict mode returns `partially_comparable` but still never guesses a value. Exact mismatches produce `not_comparable` and cite both evidence locators. ### Resource limits - 1-1,000 claims and up to 200 comparison groups. - Claims JSON is limited to 5 MB and 20 levels of nesting. - Custom aliases are limited to 500 exact entries. - Launch memory: 256 MB minimum, 1,024 MB default, 2,048 MB maximum. - Hosted timeout: 600 seconds. ### Data handling and privacy Claims are written to the default dataset and rule artifacts to the run key-value store under the account's Apify retention settings. The Actor makes no network requests in version 1. Remove run storage in Apify when it is no longer needed. Do not place credentials or private tokens in source URLs or extension fields. ### Source, rights, and attribution The controlled default claim points to a Flintglade-authored public fixture. The Actor records user-provided evidence locators and does not assert ownership, licensing, benchmark validity, or permission to reuse third-party material. ### Limitations Aliases apply only when an exact mapping exists in the selected registry. Percentage/fraction conversion occurs only for a metric whose registry definition explicitly authorizes it. Unknown metrics and unsupported units are preserved and refused, not inferred. Reported values are not fact-checked. ### Support Email `support@flintglade.com` with the run ID, registry version, and a redacted minimal claim. Never send credentials or private evidence in a support request. ### Changelog See [CHANGELOG.md](CHANGELOG.md) for release history. # Actor input Schema ## `inputSchemaVersion` (type: `integer`): Versioned runtime input contract. ## `claims` (type: `array`): Each claim must identify its evidence source, model, benchmark, and reported metric. Values remain strings so Decimal precision and operators are preserved. ## `comparisonGroups` (type: `array`): Optional claim-ID groups. Pairwise decisions and exact field evidence are emitted without rankings or extra charges. ## `aliasRegistryVersion` (type: `string`): Exact bundled registry version; aliases are never inferred. ## `customAliases` (type: `object`): Optional domain-to-alias mappings. Collisions and unknown canonical benchmark or metric targets are rejected. ## `strictMode` (type: `boolean`): When true, missing registry-required context produces insufficient\_context. When false, it remains a visible partial comparison and is never guessed. ## Actor input object example ```json { "inputSchemaVersion": 1, "claims": [ { "claimId": "demo-1", "source": { "url": "https://flintglade-tools.github.io/apify-fixtures/benchmarks/demo.html", "locator": "#claim-1" }, "model": { "name": "ExampleModel", "revision": "abc123" }, "benchmark": { "name": "ExampleBench", "version": "1.0", "split": "test" }, "metric": { "name": "accuracy", "value": "0.812", "unit": "fraction", "direction": "higher_is_better" }, "protocol": { "shots": 0, "promptTemplate": "fixture-v1" } } ], "comparisonGroups": [], "aliasRegistryVersion": "1.0.0", "customAliases": {}, "strictMode": true } ``` # Actor output Schema ## `defaultDataset` (type: `string`): No description ## `aliasRegistry` (type: `string`): No description ## `comparabilityRules` (type: `string`): No description ## `artifactManifest` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "claims": [ { "claimId": "demo-1", "source": { "url": "https://flintglade-tools.github.io/apify-fixtures/benchmarks/demo.html", "locator": "#claim-1" }, "model": { "name": "ExampleModel", "revision": "abc123" }, "benchmark": { "name": "ExampleBench", "version": "1.0", "split": "test" }, "metric": { "name": "accuracy", "value": "0.812", "unit": "fraction", "direction": "higher_is_better" }, "protocol": { "shots": 0, "promptTemplate": "fixture-v1" } } ] }; // Run the Actor and wait for it to finish const run = await client.actor("flintglade/ai-benchmark-claim-normalizer").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "claims": [{ "claimId": "demo-1", "source": { "url": "https://flintglade-tools.github.io/apify-fixtures/benchmarks/demo.html", "locator": "#claim-1", }, "model": { "name": "ExampleModel", "revision": "abc123", }, "benchmark": { "name": "ExampleBench", "version": "1.0", "split": "test", }, "metric": { "name": "accuracy", "value": "0.812", "unit": "fraction", "direction": "higher_is_better", }, "protocol": { "shots": 0, "promptTemplate": "fixture-v1", }, }] } # Run the Actor and wait for it to finish run = client.actor("flintglade/ai-benchmark-claim-normalizer").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "claims": [ { "claimId": "demo-1", "source": { "url": "https://flintglade-tools.github.io/apify-fixtures/benchmarks/demo.html", "locator": "#claim-1" }, "model": { "name": "ExampleModel", "revision": "abc123" }, "benchmark": { "name": "ExampleBench", "version": "1.0", "split": "test" }, "metric": { "name": "accuracy", "value": "0.812", "unit": "fraction", "direction": "higher_is_better" }, "protocol": { "shots": 0, "promptTemplate": "fixture-v1" } } ] }' | apify call flintglade/ai-benchmark-claim-normalizer --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=flintglade/ai-benchmark-claim-normalizer", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI Benchmark Claim Normalizer", "description": "Normalize structured AI benchmark claims and produce evidence-backed comparability refusals when versions, splits, metrics, prompts, shots, or model settings do not align.", "version": "0.1", "x-build-id": "1ZnXYjFS6Zpjy8xxu" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/flintglade~ai-benchmark-claim-normalizer/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-flintglade-ai-benchmark-claim-normalizer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/flintglade~ai-benchmark-claim-normalizer/runs": { "post": { "operationId": "runs-sync-flintglade-ai-benchmark-claim-normalizer", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/flintglade~ai-benchmark-claim-normalizer/run-sync": { "post": { "operationId": "run-sync-flintglade-ai-benchmark-claim-normalizer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "claims" ], "properties": { "inputSchemaVersion": { "title": "Input schema version", "minimum": 1, "maximum": 1, "type": "integer", "description": "Versioned runtime input contract.", "default": 1 }, "claims": { "title": "Structured benchmark claims", "minItems": 1, "maxItems": 1000, "type": "array", "description": "Each claim must identify its evidence source, model, benchmark, and reported metric. Values remain strings so Decimal precision and operators are preserved.", "items": { "type": "object", "properties": { "claimId": { "title": "Claim ID", "description": "Unique ID used by comparison groups.", "type": "string", "minLength": 1, "maxLength": 300 }, "source": { "title": "Evidence source", "description": "User-supplied evidence URL and record locator; the URL is not fetched.", "type": "object", "properties": { "url": { "title": "Evidence URL", "description": "HTTP(S) locator with no embedded credentials.", "type": "string", "maxLength": 2048 }, "locator": { "title": "Evidence locator", "description": "Section, row, or fragment label in the source evidence.", "type": "string", "minLength": 1, "maxLength": 1000 } }, "required": [ "url", "locator" ] }, "model": { "title": "Model", "description": "Reported model identity and comparison-relevant configuration.", "type": "object", "properties": { "name": { "title": "Model name", "description": "Raw reported model name; different models may be compared.", "type": "string", "minLength": 1, "maxLength": 300 }, "revision": { "title": "Model revision", "description": "Optional model revision preserved as evidence.", "type": "string", "maxLength": 300 }, "precision": { "title": "Precision", "description": "Reported numeric precision such as fp16.", "type": "string", "maxLength": 100 }, "quantization": { "title": "Quantization", "description": "Reported quantization setting such as none or int8.", "type": "string", "maxLength": 100 }, "contextLength": { "title": "Context length", "description": "Reported model context length.", "type": "integer", "minimum": 1, "maximum": 100000000 } }, "required": [ "name" ] }, "benchmark": { "title": "Benchmark", "description": "Benchmark, dataset, version, and split reported by the source.", "type": "object", "properties": { "name": { "title": "Benchmark name", "description": "Raw benchmark name subject only to exact aliases.", "type": "string", "minLength": 1, "maxLength": 300 }, "dataset": { "title": "Dataset", "description": "Optional explicit dataset name.", "type": "string", "maxLength": 300 }, "version": { "title": "Benchmark version", "description": "Comparison-critical benchmark version.", "type": "string", "maxLength": 200 }, "datasetVersion": { "title": "Dataset version", "description": "Optional dataset release or revision.", "type": "string", "maxLength": 200 }, "split": { "title": "Dataset split", "description": "Comparison-critical split such as test or validation.", "type": "string", "maxLength": 100 } }, "required": [ "name" ] }, "metric": { "title": "Metric", "description": "Reported metric semantics and Decimal-preserving value.", "type": "object", "properties": { "name": { "title": "Metric name", "description": "Raw metric name subject only to exact aliases.", "type": "string", "minLength": 1, "maxLength": 300 }, "value": { "title": "Reported value", "description": "Finite decimal string with an optional comparison operator.", "type": "string", "minLength": 1, "maxLength": 200 }, "unit": { "title": "Metric unit", "description": "Reported unit; conversions require an explicit registry rule.", "type": "string", "maxLength": 100 }, "direction": { "title": "Metric direction", "description": "Whether higher or lower reported values are better.", "type": "string", "enum": [ "higher_is_better", "lower_is_better" ] }, "confidenceInterval": { "title": "Confidence interval", "description": "Optional ordered closed interval preserved as Decimal strings.", "type": "object", "additionalProperties": false, "properties": { "lower": { "title": "Lower bound", "description": "Finite Decimal lower bound.", "type": "string", "minLength": 1, "maxLength": 200 }, "upper": { "title": "Upper bound", "description": "Finite Decimal upper bound.", "type": "string", "minLength": 1, "maxLength": 200 }, "level": { "title": "Confidence level", "description": "Reported confidence level label.", "type": "string", "maxLength": 100 } }, "required": [ "lower", "upper" ] }, "uncertainty": { "title": "Reported uncertainty", "description": "Optional source-provided uncertainty description.", "type": "string", "maxLength": 500 } }, "required": [ "name", "value" ] }, "protocol": { "title": "Evaluation protocol", "description": "Prompting, shots, decoding, and hardware context.", "type": "object", "properties": { "shots": { "title": "Shots", "description": "Number of examples supplied to the model.", "type": "integer", "minimum": 0, "maximum": 1000000 }, "promptTemplate": { "title": "Prompt template", "description": "Stable prompt or template identifier.", "type": "string", "maxLength": 20000 }, "decoding": { "title": "Decoding settings", "description": "Reported decoding configuration.", "type": "string", "maxLength": 2000 }, "hardware": { "title": "Hardware", "description": "Reported evaluation hardware.", "type": "string", "maxLength": 2000 } } }, "date": { "title": "Claim date", "description": "Optional source-reported date excluded from the context fingerprint.", "type": "string", "maxLength": 100 }, "evidenceExcerpt": { "title": "Evidence excerpt", "description": "Optional bounded excerpt supplied by the user.", "type": "string", "maxLength": 20000 } }, "required": [ "claimId", "source", "model", "benchmark", "metric" ] } }, "comparisonGroups": { "title": "Explicit comparison groups", "maxItems": 200, "type": "array", "description": "Optional claim-ID groups. Pairwise decisions and exact field evidence are emitted without rankings or extra charges.", "items": { "type": "object", "additionalProperties": false, "properties": { "groupId": { "title": "Group ID", "description": "Unique label for this comparison group.", "type": "string", "minLength": 1, "maxLength": 300 }, "claimIds": { "title": "Claim IDs", "description": "Two to fifty unique claim IDs from this run.", "type": "array", "minItems": 2, "maxItems": 50, "items": { "type": "string", "minLength": 1, "maxLength": 300 } } }, "required": [ "groupId", "claimIds" ] }, "default": [] }, "aliasRegistryVersion": { "title": "Alias registry version", "enum": [ "1.0.0" ], "type": "string", "description": "Exact bundled registry version; aliases are never inferred.", "default": "1.0.0" }, "customAliases": { "title": "Explicit custom aliases", "type": "object", "description": "Optional domain-to-alias mappings. Collisions and unknown canonical benchmark or metric targets are rejected.", "default": {} }, "strictMode": { "title": "Strict comparison context", "type": "boolean", "description": "When true, missing registry-required context produces insufficient_context. When false, it remains a visible partial comparison and is never guessed.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```