# ToolDiff: MCP Schema Snapshot and Diff (`savagnic/tooldiff`) Actor

Snapshot an MCP server's tools/list and detect breaking, additive, and neutral schema changes. SSRF-hardened and deterministic.

- **URL**: https://apify.com/savagnic/tooldiff.md
- **Developed by:** [NICHOLAS SAVAGE](https://apify.com/savagnic) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$50.00 / 1,000 successful results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## ToolDiff — MCP `tools/list` snapshot and semantic schema diff

ToolDiff takes a snapshot of what an MCP server advertises over real JSON-RPC 2.0
HTTP, and compares two snapshots **semantically**: every change is classified as
**breaking**, **additive** or **neutral** for a caller.

If a server you depend on quietly adds a required parameter, drops an enum
value or withdraws a `readOnlyHint`, your agent starts failing in production.
ToolDiff turns that into a change gate you can run in CI or on a schedule.

### What it does

| Mode | What you give it | What you get |
|---|---|---|
| `snapshot` | a `url` | the server's advertised tool set, normalized, with a content id |
| `diff` | a `beforeSnapshot` plus either an `afterSnapshot` or a `url` | every change, each labelled breaking / additive / neutral, plus a summary |

Set `failOnBreaking` and the run exits `1` when the diff contains a breaking
change — the result is still stored, so the run is readable either way.

### Input

Full contract in `input_schema.json`. Unknown fields are **rejected**, not
ignored; that is what keeps the network policy unreachable from input.

| Field | Type | Default | Notes |
|---|---|---|---|
| `mode` | `snapshot` | `diff` | `snapshot` | required |
| `url` | string | – | required in `snapshot` mode; in `diff` mode it supplies the "after" side |
| `beforeSnapshot` | object | – | required in `diff` mode |
| `afterSnapshot` | object | – | `diff` mode needs this or `url` |
| `failOnBreaking` | boolean | `false` | exit `1` when the diff is breaking |
| `allowedHosts` | string\[] | `[]` | narrows the reachable set; it can never widen it |
| `timeoutMs` | integer | `20000` | 1000–120000 |
| `maxTools` | integer | `5000` | 1–100000 |
| `authorizationHeader` | string (secret) | – | forwarded to the MCP server as `Authorization`, never stored or logged |
| `tenantId` | string | – | echoed back for correlation |

### Output

One `wsb.result/1` envelope per run, written **identically** to the default
dataset and to the `OUTPUT` key-value record, on success and on failure alike.
The shape is described in `dataset_schema.json` and `output_schema.json`; a real
run's output is in [`sample-output.json`](./sample-output.json), produced by

```bash
npm ci && npm run build
node products/tooldiff/dist/actor/main.js   # with the two bundled fixtures as input
```

so you can reproduce it byte for byte apart from the timestamp and duration.

Exit codes:

| Code | Meaning |
|---|---|
| `0` | ran, result stored |
| `1` | `failOnBreaking` was set and the diff is breaking (the result is still stored) |
| `2` | input missing or invalid |
| `3` | runtime, network or storage failure |

### Pricing

Pay per event, one event:

| Event | Price | Charged when |
|---|---|---|
| `successful-result` | **$0.05** | once, after a successful result has been written to both the dataset and `OUTPUT` |

Nothing else is billable. In particular you are **not** charged for:

- a rejected or invalid input (exit `2`),
- a runtime, network or upstream failure, including an SSRF-blocked target (exit `3`),
- a run whose `failOnBreaking` gate tripped (exit `1`) — a breaking change is
  news you asked for, and the run is a failed run,
- a run whose result could not be stored,
- a start, a retry or a resurrect.

The charge is issued once per run, with an idempotency key, only after the
result is durably stored, and the run's exit code never depends on whether the
billing call succeeded. Your maximum cost per run is respected before any charge
is sent.

### Security and network policy

ToolDiff fetches a URL you supply, so it is written for that threat model:

- `http` and `https` only; URL credentials rejected; redirects never followed.
- Every DNS answer is checked against blocked ranges **before** connecting —
  loopback, RFC 1918, carrier-grade NAT, link-local (including the cloud
  metadata address `169.254.169.254`), reserved and multicast space, IPv6 ULA
  and link-local, and IPv4-mapped forms of all of them. A hostname resolving to
  both a public and a private address is refused outright.
- The connection is pinned to the verified address by overriding name
  resolution, not by rewriting the URL, so the `Host` header, TLS SNI and
  certificate validation all still use the real hostname. Certificate
  verification cannot be turned off.
- The policy is fixed in code. No input field relaxes it, and unknown input
  fields are rejected so a new one cannot be smuggled in.
- `authorizationHeader` is sent to the MCP server and nowhere else. It is
  replaced with `[redacted]` in the stored input echo, and a fail-closed guard
  scans the serialized result for it before anything is written.

### Boundaries

- It compares what a server **advertises**. It never calls a tool, so it cannot
  see a behaviour change behind an unchanged schema.
- Classification is deliberately conservative: where equivalence cannot be
  proved — two regexes, two `anyOf` branches — the change is reported as
  breaking.
- JSON Schema `$ref`, `$defs` and remote references are not resolved.
- A snapshot is one server at one moment. Servers may advertise different tools
  per session, per authenticated identity or behind feature flags.
- The session-based Streamable HTTP flow of MCP 2024-11-05, 2025-03-26 and
  2025-06-18 is implemented. The stateless 2026-07-28 revision is not; only its
  required `Mcp-Method` request header is sent.
- Breaking-change classification is this product's opinion of MCP schema
  semantics, not a Model Context Protocol specification ruling.
- Not affiliated with or endorsed by the Model Context Protocol maintainers or
  by any MCP server operator whose endpoint you point it at.

### Provenance

The last build with live evidence behind it is Apify build `1.2.3`
(`75lrmHaZGr4K2Qw8W`) from source commit `91dbfc2`. Against
`https://mcp.apify.com` that build negotiated protocol `2025-06-18` with
`apify-mcp-server` `0.14.2`, established a session and snapshotted 4 tools; a
run pointed at the cloud metadata address was blocked with `NETWORK_BLOCKED` and
exit `3`; and a secret-canary run left no trace of the canary in the dataset, the
`OUTPUT` record or the log. Each of those runs wrote a dataset item and an
`OUTPUT` record that matched exactly. The full record, including what that
evidence does *not* cover, is in `docs/evidence/tooldiff-apify-live-2026-08-11.md`.

### Licence

Apache-2.0.

# Actor input Schema

## `mode` (type: `string`):

Choose whether to capture a live MCP tools/list snapshot or compare a baseline against another snapshot or live endpoint.

## `url` (type: `string`):

http or https only. Loopback, RFC 1918, carrier-grade NAT and link-local addresses (including the 169.254.169.254 cloud metadata endpoint) are refused.

## `beforeSnapshot` (type: `object`):

A previously captured snapshot, a raw tools/list result, or an object with a `tools` array.

## `afterSnapshot` (type: `object`):

Optional. When omitted in diff mode, the live URL is snapshotted instead.

## `failOnBreaking` (type: `boolean`):

Use this to turn the actor into a CI change gate.

## `allowedHosts` (type: `array`):

Optional hostname allowlist. When set, only these hosts (and their subdomains) may be contacted.

## `timeoutMs` (type: `integer`):

Wall-clock budget for the whole snapshot including all pagination.

## `maxTools` (type: `integer`):

Refuse servers advertising more tools than this.

## `authorizationHeader` (type: `string`):

Optional. Supplied by you at run time and never stored by this actor. Example: "Bearer eyJ...".

## `tenantId` (type: `string`):

Optional operator-defined identifier forwarded to usage events for attribution.

## Actor input object example

```json
{
  "mode": "snapshot",
  "url": "https://mcp.example.com/mcp",
  "failOnBreaking": false,
  "timeoutMs": 20000,
  "maxTools": 5000
}
```

# Actor output Schema

## `result` (type: `string`):

The single result envelope as a dataset item: the snapshot, the semantic diff and the breaking-change gate outcome.

## `outputRecord` (type: `string`):

The identical envelope in the default key-value store under the OUTPUT key.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("savagnic/tooldiff").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("savagnic/tooldiff").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call savagnic/tooldiff --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,savagnic/tooldiff"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XnEwBQiGropSzXXSY/builds/ldYZDvweijIwrklHs/openapi.json