MCP Proof — bounded MCP evaluation with an optional signed run receipt

Point MCP Proof at a Model Context Protocol server, give it your known-answer cases, and get back one machine-readable record of what that server actually did: which tools it advertised, what their schemas looked like at that moment, and which of your cases passed, failed or errored.

This is not certification. No programme, body or standard stands behind the output. It is a run record: one server, one moment, one set of cases you wrote.

What it does

Step	What happens
Resolve	The endpoint is resolved and every returned address is checked before a socket is opened.
Handshake	A real MCP initialize over Streamable HTTP, then tools/list with pagination followed to a bounded page limit.
Snapshot	The advertised tool set is canonicalised and hashed, so two runs of the same server are comparable byte for byte.
Inspect	The tool schemas are checked against a published, closed rule set (MP-SCH-001 … MP-SCH-010). Findings are observations, not defects.
Evaluate	Each of your cases is executed as a real tools/call and graded with one deterministic matcher.
Record	One bounded result is written, optionally with a signed receipt.

Seven matchers are available per case: textEquals, textContainsAll, textExcludes, textMatches, jsonPointerEquals, jsonPointerExists and isError. A case that times out or errors is recorded as errored, never silently as a failure.

Input

Field	Type	Default	Notes
url	string	—	Required. The Streamable HTTP MCP endpoint. http and https only.
evaluationSet	object	—	Required. Your cases, schema mcp-proof/evaluation-set/1.
sign	boolean	false	Emit a signed run receipt. Refused unless the operator provisioned a keystore — see Signing below.
failOnCaseFailure	boolean	false	Exit 1 when any case does not pass. Turns the run into a CI gate.
allowedHosts	array	—	Narrows the hostnames this run may reach. It can only narrow.
caseTimeoutMs	integer	20000	Per-tools/call budget, 1000–120000.
maxCases	integer	500	A larger set is refused, never truncated. 1–500.
timeoutMs	integer	20000	Budget for the initialize + tools/list handshake, 1000–120000.
maxTools	integer	5000	A server advertising more is refused. 1–100000.
authorizationHeader	string (secret)	—	Sent to the MCP server as Authorization and nowhere else. Redacted everywhere it is echoed.
tenantId	string	—	Echoed back for correlation. Nothing else uses it.

Any field not in this table is rejected with INPUT_INVALID, not ignored. That is what makes the network policy non-bypassable from input.

Output

A run writes exactly one result, and writes it to both places:

• one item in the default dataset, and
• the OUTPUT record in the default key-value store.

The two are the same bytes, on success and on failure alike, so a failed run still leaves a readable record. The envelope is wsb.result/1 and the payload is mcp-proof/actor-result/1: the redacted input echo, the evaluation report (server, evaluation-set hash, snapshot hash, graded cases, schema findings, metrics), the signing outcome, the receipt and public key when one was produced, and the case-gate state.

If a very large run would exceed the 4 MB result ceiling, the receipt is dropped first and then the per-case detail, and every reduction is recorded as a warning in the envelope. Nothing is truncated quietly.

sample-output.json in this directory is a real result from a local offline run against the bundled fixture server. It is labelled as such inside the file, and it is not evidence of a platform run.

Exit codes

Code	Meaning
0	The evaluation completed. Cases may still have failed; check data.report.metrics.
1	failOnCaseFailure was set and not every case passed.
2	The input or the runtime configuration was rejected. Retrying unchanged will not help.
3	A runtime, protocol, network-policy or storage failure. May be transient.

Pricing

Event	Price	When
evaluation-run	$0.20	once, after a successful result has been written to both the dataset and OUTPUT

Not charged: a rejected input, a refused signing request, a blocked address, a protocol or upstream failure, a tripped failOnCaseFailure gate, a tripped secret guard, and any run whose result could not be stored. The charge is made after storage, never before, and carries an idempotency key so a retry cannot double-bill.

The price above is a recommendation recorded in this repository. Nothing is configured in the Apify Console and no charge has ever been made by this Actor.

Signing

sign: true produces an Ed25519-signed run receipt binding the evaluation-set hash, the snapshot hash and the metrics to a timestamp. The run will refuse to sign unless the operator provisioned a keystore through the SAVAGE_RECEIPT_KEYSTORE environment variable: signing with a key the run invented for itself would produce a receipt nobody can attribute, which looks like assurance and is not.

Even with a keystore, this is file-based development signing, not production key custody. There is no HSM, no KMS, no key escrow, no rotation policy and no revocation transparency log. A receipt is a run record, not a certificate.

Security and network policy

• http and https only. URLs carrying credentials are refused.
• Every resolved address is checked, not just the first: loopback, RFC 1918, carrier-grade NAT, link-local (including 169.254.169.254), reserved and multicast ranges are refused, and a DNS answer mixing public and private addresses is refused outright.
• The connection is pinned to the vetted address while keeping the real hostname for SNI and certificate verification. TLS verification is never disabled.
• Redirects are never followed.
• Response bodies, page counts and tool counts are all bounded.
• authorizationHeader is sent to the target server and nowhere else. A fail-closed guard scans the serialised result for it before anything is written, and the run fails rather than storing a leak.

Boundaries

Every result repeats these in boundaries:

• Not certification, accreditation, conformance testing or approval.
• Covers exactly the supplied evaluation set and exactly the snapshot taken at the recorded moment.
• Not a security assessment: no penetration testing, no authentication testing, no prompt-injection testing, no supply-chain review.
• Schema findings are observations against a closed rule set, not defects, vulnerabilities, quality scores or specification violations.
• A pass means one deterministic matcher held once.
• Re-running later may produce a different result. Servers change without notice.
• Signer-key custody is an unresolved prerequisite.

Provenance

The SSRF policy, the pinning transport, the Actor storage layer and the charging layer are vendored copies of the equivalent modules in the ToolDiff product in the same repository, under the same licence and by the same author. They are copied rather than imported so this product stays a self-contained deployable directory. A test asserts the vendored transport has not drifted from its source.

Licence

Apache-2.0. See LICENSE and NOTICE.md at the repository root.