ToolDiff — MCP tools/list snapshot and semantic schema diff

ToolDiff takes a snapshot of what an MCP server advertises over real JSON-RPC 2.0 HTTP, and compares two snapshots semantically: every change is classified as breaking, additive or neutral for a caller.

If a server you depend on quietly adds a required parameter, drops an enum value or withdraws a readOnlyHint, your agent starts failing in production. ToolDiff turns that into a change gate you can run in CI or on a schedule.

What it does

Mode	What you give it	What you get
snapshot	a url	the server's advertised tool set, normalized, with a content id
diff	a beforeSnapshot plus either an afterSnapshot or a url	every change, each labelled breaking / additive / neutral, plus a summary

Set failOnBreaking and the run exits 1 when the diff contains a breaking change — the result is still stored, so the run is readable either way.

Input

Full contract in input_schema.json. Unknown fields are rejected, not ignored; that is what keeps the network policy unreachable from input.

Field	Type	Default	Notes
mode	snapshot | diff	snapshot	required
url	string	–	required in snapshot mode; in diff mode it supplies the "after" side
beforeSnapshot	object	–	required in diff mode
afterSnapshot	object	–	diff mode needs this or url
failOnBreaking	boolean	false	exit 1 when the diff is breaking
allowedHosts	string[]	[]	narrows the reachable set; it can never widen it
timeoutMs	integer	20000	1000–120000
maxTools	integer	5000	1–100000
authorizationHeader	string (secret)	–	forwarded to the MCP server as Authorization, never stored or logged
tenantId	string	–	echoed back for correlation

Output

One wsb.result/1 envelope per run, written identically to the default dataset and to the OUTPUT key-value record, on success and on failure alike. The shape is described in dataset_schema.json and output_schema.json; a real run's output is in ./sample-output.json, produced by

npm ci && npm run build
node products/tooldiff/dist/actor/main.js   # with the two bundled fixtures as input

so you can reproduce it byte for byte apart from the timestamp and duration.

Exit codes:

Code	Meaning
0	ran, result stored
1	failOnBreaking was set and the diff is breaking (the result is still stored)
2	input missing or invalid
3	runtime, network or storage failure

Pricing

Pay per event, one event:

Event	Price	Charged when
successful-result	$0.05	once, after a successful result has been written to both the dataset and OUTPUT

Nothing else is billable. In particular you are not charged for:

• a rejected or invalid input (exit 2),
• a runtime, network or upstream failure, including an SSRF-blocked target (exit 3),
• a run whose failOnBreaking gate tripped (exit 1) — a breaking change is news you asked for, and the run is a failed run,
• a run whose result could not be stored,
• a start, a retry or a resurrect.

The charge is issued once per run, with an idempotency key, only after the result is durably stored, and the run's exit code never depends on whether the billing call succeeded. Your maximum cost per run is respected before any charge is sent.

Security and network policy

ToolDiff fetches a URL you supply, so it is written for that threat model:

• http and https only; URL credentials rejected; redirects never followed.
• Every DNS answer is checked against blocked ranges before connecting — loopback, RFC 1918, carrier-grade NAT, link-local (including the cloud metadata address 169.254.169.254), reserved and multicast space, IPv6 ULA and link-local, and IPv4-mapped forms of all of them. A hostname resolving to both a public and a private address is refused outright.
• The connection is pinned to the verified address by overriding name resolution, not by rewriting the URL, so the Host header, TLS SNI and certificate validation all still use the real hostname. Certificate verification cannot be turned off.
• The policy is fixed in code. No input field relaxes it, and unknown input fields are rejected so a new one cannot be smuggled in.
• authorizationHeader is sent to the MCP server and nowhere else. It is replaced with [redacted] in the stored input echo, and a fail-closed guard scans the serialized result for it before anything is written.

Boundaries

• It compares what a server advertises. It never calls a tool, so it cannot see a behaviour change behind an unchanged schema.
• Classification is deliberately conservative: where equivalence cannot be proved — two regexes, two anyOf branches — the change is reported as breaking.
• JSON Schema $ref, $defs and remote references are not resolved.
• A snapshot is one server at one moment. Servers may advertise different tools per session, per authenticated identity or behind feature flags.
• The session-based Streamable HTTP flow of MCP 2024-11-05, 2025-03-26 and 2025-06-18 is implemented. The stateless 2026-07-28 revision is not; only its required Mcp-Method request header is sent.
• Breaking-change classification is this product's opinion of MCP schema semantics, not a Model Context Protocol specification ruling.
• Not affiliated with or endorsed by the Model Context Protocol maintainers or by any MCP server operator whose endpoint you point it at.

Provenance

The last build with live evidence behind it is Apify build 1.2.3 (75lrmHaZGr4K2Qw8W) from source commit 91dbfc2. Against https://mcp.apify.com that build negotiated protocol 2025-06-18 with apify-mcp-server 0.14.2, established a session and snapshotted 4 tools; a run pointed at the cloud metadata address was blocked with NETWORK_BLOCKED and exit 3; and a secret-canary run left no trace of the canary in the dataset, the OUTPUT record or the log. Each of those runs wrote a dataset item and an OUTPUT record that matched exactly. The full record, including what that evidence does not cover, is in docs/evidence/tooldiff-apify-live-2026-08-11.md.

Licence

Apache-2.0.