# MCP Proof: Bounded MCP Evaluation (`savagnic/mcp-proof`) Actor

Run a bounded known-answer evaluation set against an MCP server, snapshot tool schemas, grade deterministically, and return an evidence receipt. Not certification.

- **URL**: https://apify.com/savagnic/mcp-proof.md
- **Developed by:** [NICHOLAS SAVAGE](https://apify.com/savagnic) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## MCP Proof — bounded MCP evaluation with an optional signed run receipt

Point MCP Proof at a Model Context Protocol server, give it **your** known-answer
cases, and get back one machine-readable record of what that server actually did:
which tools it advertised, what their schemas looked like at that moment, and
which of your cases passed, failed or errored.

**This is not certification.** No programme, body or standard stands behind the
output. It is a run record: one server, one moment, one set of cases you wrote.

### What it does

| Step | What happens |
| --- | --- |
| Resolve | The endpoint is resolved and every returned address is checked before a socket is opened. |
| Handshake | A real MCP `initialize` over Streamable HTTP, then `tools/list` with pagination followed to a bounded page limit. |
| Snapshot | The advertised tool set is canonicalised and hashed, so two runs of the same server are comparable byte for byte. |
| Inspect | The tool schemas are checked against a published, closed rule set (`MP-SCH-001` … `MP-SCH-010`). Findings are observations, not defects. |
| Evaluate | Each of your cases is executed as a real `tools/call` and graded with one deterministic matcher. |
| Record | One bounded result is written, optionally with a signed receipt. |

Seven matchers are available per case: `textEquals`, `textContainsAll`,
`textExcludes`, `textMatches`, `jsonPointerEquals`, `jsonPointerExists` and
`isError`. A case that times out or errors is recorded as **errored**, never
silently as a failure.

### Input

| Field | Type | Default | Notes |
| --- | --- | --- | --- |
| `url` | string | — | Required. The Streamable HTTP MCP endpoint. `http` and `https` only. |
| `evaluationSet` | object | — | Required. Your cases, schema `mcp-proof/evaluation-set/1`. |
| `sign` | boolean | `false` | Emit a signed run receipt. Refused unless the operator provisioned a keystore — see **Signing** below. |
| `failOnCaseFailure` | boolean | `false` | Exit `1` when any case does not pass. Turns the run into a CI gate. |
| `allowedHosts` | array | — | Narrows the hostnames this run may reach. It can only narrow. |
| `caseTimeoutMs` | integer | `20000` | Per-`tools/call` budget, 1000–120000. |
| `maxCases` | integer | `500` | A larger set is refused, never truncated. 1–500. |
| `timeoutMs` | integer | `20000` | Budget for the `initialize` + `tools/list` handshake, 1000–120000. |
| `maxTools` | integer | `5000` | A server advertising more is refused. 1–100000. |
| `authorizationHeader` | string (secret) | — | Sent to the MCP server as `Authorization` and nowhere else. Redacted everywhere it is echoed. |
| `tenantId` | string | — | Echoed back for correlation. Nothing else uses it. |

Any field not in this table is **rejected** with `INPUT_INVALID`, not ignored.
That is what makes the network policy non-bypassable from input.

### Output

A run writes **exactly one** result, and writes it to **both** places:

- one item in the default dataset, and
- the `OUTPUT` record in the default key-value store.

The two are the same bytes, on success and on failure alike, so a failed run
still leaves a readable record. The envelope is `wsb.result/1` and the payload
is `mcp-proof/actor-result/1`: the redacted input echo, the evaluation report
(server, evaluation-set hash, snapshot hash, graded cases, schema findings,
metrics), the signing outcome, the receipt and public key when one was produced,
and the case-gate state.

If a very large run would exceed the 4 MB result ceiling, the receipt is dropped
first and then the per-case detail, and **every** reduction is recorded as a
warning in the envelope. Nothing is truncated quietly.

`sample-output.json` in this directory is a real result from a local offline run
against the bundled fixture server. It is labelled as such inside the file, and
it is not evidence of a platform run.

### Exit codes

| Code | Meaning |
| --- | --- |
| `0` | The evaluation completed. Cases may still have failed; check `data.report.metrics`. |
| `1` | `failOnCaseFailure` was set and not every case passed. |
| `2` | The input or the runtime configuration was rejected. Retrying unchanged will not help. |
| `3` | A runtime, protocol, network-policy or storage failure. May be transient. |

### Pricing

| Event | Price | When |
| --- | --- | --- |
| `evaluation-run` | **$0.20** | once, after a successful result has been written to both the dataset and `OUTPUT` |

Not charged: a rejected input, a refused signing request, a blocked address, a
protocol or upstream failure, a tripped `failOnCaseFailure` gate, a tripped
secret guard, and any run whose result could not be stored. The charge is made
after storage, never before, and carries an idempotency key so a retry cannot
double-bill.

The price above is a recommendation recorded in this repository. **Nothing is
configured in the Apify Console and no charge has ever been made by this Actor.**

### Signing

`sign: true` produces an Ed25519-signed run receipt binding the evaluation-set
hash, the snapshot hash and the metrics to a timestamp. The run will **refuse**
to sign unless the operator provisioned a keystore through the
`SAVAGE_RECEIPT_KEYSTORE` environment variable: signing with a key the run
invented for itself would produce a receipt nobody can attribute, which looks
like assurance and is not.

Even with a keystore, this is **file-based development signing, not production
key custody**. There is no HSM, no KMS, no key escrow, no rotation policy and no
revocation transparency log. A receipt is a run record, not a certificate.

### Security and network policy

- `http` and `https` only. URLs carrying credentials are refused.
- Every resolved address is checked, not just the first: loopback, RFC 1918,
  carrier-grade NAT, link-local (including `169.254.169.254`), reserved and
  multicast ranges are refused, and a DNS answer mixing public and private
  addresses is refused outright.
- The connection is pinned to the vetted address while keeping the real hostname
  for SNI and certificate verification. TLS verification is never disabled.
- Redirects are never followed.
- Response bodies, page counts and tool counts are all bounded.
- `authorizationHeader` is sent to the target server and nowhere else. A
  fail-closed guard scans the serialised result for it before anything is
  written, and the run fails rather than storing a leak.

### Boundaries

Every result repeats these in `boundaries`:

- Not certification, accreditation, conformance testing or approval.
- Covers exactly the supplied evaluation set and exactly the snapshot taken at
  the recorded moment.
- Not a security assessment: no penetration testing, no authentication testing,
  no prompt-injection testing, no supply-chain review.
- Schema findings are observations against a closed rule set, not defects,
  vulnerabilities, quality scores or specification violations.
- A pass means one deterministic matcher held once.
- Re-running later may produce a different result. Servers change without notice.
- Signer-key custody is an unresolved prerequisite.

### Provenance

The SSRF policy, the pinning transport, the Actor storage layer and the charging
layer are vendored copies of the equivalent modules in the ToolDiff product in
the same repository, under the same licence and by the same author. They are
copied rather than imported so this product stays a self-contained deployable
directory. A test asserts the vendored transport has not drifted from its source.

### Licence

Apache-2.0. See `LICENSE` and `NOTICE.md` at the repository root.

# Actor input Schema

## `url` (type: `string`):

Streamable HTTP MCP endpoint to evaluate. http and https only. Loopback, RFC 1918, carrier-grade NAT, link-local (including the cloud metadata address 169.254.169.254), reserved and multicast addresses are refused before any connection is made, and redirects are never followed.

## `evaluationSet` (type: `object`):

Your known-answer cases. MCP Proof executes exactly these and claims nothing wider. Each case names a tool, its arguments and one deterministic matcher: textEquals, textContainsAll, textExcludes, textMatches, jsonPointerEquals, jsonPointerExists or isError.

## `sign` (type: `boolean`):

Off by default and refused unless the operator provisioned a keystore through SAVAGE\_RECEIPT\_KEYSTORE. A run will not sign with a key it invented for itself, because a receipt nobody can attribute is worse than no receipt. A receipt is a run record, not a certificate.

## `failOnCaseFailure` (type: `boolean`):

Turns the run into a CI gate. The result is still stored either way, and a tripped gate is never billed.

## `allowedHosts` (type: `array`):

Narrows the set of hostnames this run may reach. It can only narrow: it never widens the fixed network policy.

## `caseTimeoutMs` (type: `integer`):

Budget for one tools/call. A case that exceeds it is recorded as errored, never as a failure.

## `maxCases` (type: `integer`):

The run is refused rather than truncated when the evaluation set is larger.

## `timeoutMs` (type: `integer`):

Budget for the initialize and tools/list handshake.

## `maxTools` (type: `integer`):

The run is refused when the server advertises more tools than this.

## `authorizationHeader` (type: `string`):

Sent to the MCP server as the Authorization header and nowhere else. Replaced with \[redacted] in the stored input echo, and a fail-closed guard scans the serialized result for it before anything is written.

## `tenantId` (type: `string`):

Echoed back for correlation. Not used for anything else.

## Actor input object example

```json
{
  "url": "https://mcp.example.com/mcp",
  "evaluationSet": {
    "schema": "mcp-proof/evaluation-set/1",
    "name": "my-smoke-set",
    "version": "1.0.0",
    "expectedTools": [
      "lookup_zip"
    ],
    "cases": [
      {
        "id": "zip-97201-portland",
        "tool": "lookup_zip",
        "arguments": {
          "zip": "97201"
        },
        "expect": {
          "kind": "textContainsAll",
          "values": [
            "Portland"
          ]
        }
      }
    ]
  },
  "sign": false,
  "failOnCaseFailure": false,
  "caseTimeoutMs": 20000,
  "maxCases": 500,
  "timeoutMs": 20000,
  "maxTools": 5000
}
```

# Actor output Schema

## `result` (type: `string`):

The single result envelope as a dataset item: the evaluation report, the tool-schema snapshot hashes, the graded cases, the metrics, the signing outcome and the receipt when one was produced.

## `outputRecord` (type: `string`):

The identical envelope in the default key-value store under the OUTPUT key.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://mcp.example.com/mcp",
    "evaluationSet": {
        "schema": "mcp-proof/evaluation-set/1",
        "name": "my-smoke-set",
        "version": "1.0.0",
        "expectedTools": [
            "lookup_zip"
        ],
        "cases": [
            {
                "id": "zip-97201-portland",
                "tool": "lookup_zip",
                "arguments": {
                    "zip": "97201"
                },
                "expect": {
                    "kind": "textContainsAll",
                    "values": [
                        "Portland"
                    ]
                }
            }
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("savagnic/mcp-proof").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://mcp.example.com/mcp",
    "evaluationSet": {
        "schema": "mcp-proof/evaluation-set/1",
        "name": "my-smoke-set",
        "version": "1.0.0",
        "expectedTools": ["lookup_zip"],
        "cases": [{
                "id": "zip-97201-portland",
                "tool": "lookup_zip",
                "arguments": { "zip": "97201" },
                "expect": {
                    "kind": "textContainsAll",
                    "values": ["Portland"],
                },
            }],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("savagnic/mcp-proof").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://mcp.example.com/mcp",
  "evaluationSet": {
    "schema": "mcp-proof/evaluation-set/1",
    "name": "my-smoke-set",
    "version": "1.0.0",
    "expectedTools": [
      "lookup_zip"
    ],
    "cases": [
      {
        "id": "zip-97201-portland",
        "tool": "lookup_zip",
        "arguments": {
          "zip": "97201"
        },
        "expect": {
          "kind": "textContainsAll",
          "values": [
            "Portland"
          ]
        }
      }
    ]
  }
}' |
apify call savagnic/mcp-proof --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,savagnic/mcp-proof"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ZIMRk8zstkXgeHPJ4/builds/h1K7A1Kg2q89tpgVW/openapi.json