MCP Reliability Inspector

MCP Reliability Inspector is an MCP server linter for public GitHub repositories. It reads a TypeScript or JavaScript Model Context Protocol server, checks each tool definition and inputSchema, and compares the contract with a saved baseline.

The scan is static analysis. Babel walks the source as a syntax tree. Tool handlers, install scripts, and dependencies stay unexecuted. A finished run gives you a findings table, a full JSON report, and a baseline file for the next scan.

You can run it on Apify, then call the same input from the Apify API, a schedule, or an integration.

What does MCP Reliability Inspector do?

• Tool definitions. Names must be present, unique, and within the MCP naming rules. Descriptions must exist and be long enough for an agent to choose the tool.
• Input schemas. Each inputSchema is checked as JSON Schema. Required fields must exist in properties. Schemas built by a function call or a variable are reported as dynamic.
• Baseline diff. A previous inventory flags removed tools, schema changes, description changes, and new tools.
• Coverage notes. Repositories with no recognized MCP registration, and detections without a line number, are reported separately from defects.

Confirmed defects are deterministic findings at high or medium severity. Info results and coverage notes describe the scan. They are quality or coverage signals.

Use cases

• Before you publish an MCP server. Catch empty tool names, duplicate names, missing descriptions, and broken input schemas on the commit you are about to tag.
• CI regression check. Save the BASELINE record from a good commit and pass it on the next run. Removed tools and schema changes show up as baseline findings.
• Agent-readiness review. MCP clients pick tools from the name, description, and schema. This scan lists the definitions an agent would have to guess.

How to scan a GitHub MCP server

1. Open the Actor and start a run. The repository field is prefilled with modelcontextprotocol/servers. Replace it with the owner/repo you want.
2. Leave the default checks on, or add Baseline diff and paste a previous baseline.
3. Run the Actor. Apify stores the log, and you can schedule the same input or call it from the Apify API.
4. Open the Findings view for one row per issue. Open the Summary view for status and severity counts.
5. Save the BASELINE record from the run output if the next scan should detect contract changes.

Input

{
  "repository": "modelcontextprotocol/servers",
  "ref": "main",
  "checks": ["tool-definitions", "input-schemas"],
  "includeSourceExcerpts": true,
  "limits": {
    "maxFiles": 5000,
    "maxFileBytes": 500000,
    "maxTotalSourceBytes": 25000000
  }
}

Field	Required	Default	Purpose
repository	Yes	Prefill modelcontextprotocol/servers	Public GitHub repo as owner/repo or a https://github.com/owner/repo URL.
ref	No	Default branch	Branch name, tag, or full commit SHA. The report records the commit that was scanned.
checks	No	tool-definitions, input-schemas	tool-definitions checks names and descriptions. input-schemas checks JSON Schema. baseline-diff compares with baseline.
baseline	No	Empty	Baseline inventory from an earlier BASELINE record. On Apify, also select Baseline diff in Checks.
includeSourceExcerpts	No	true	Include a few source lines on each finding.
limits.maxFiles	No	5000	Maximum candidate files (1–10000).
limits.maxFileBytes	No	500000	Maximum bytes per file.
limits.maxTotalSourceBytes	No	25000000	Maximum source bytes across the scan.

Baseline inventory

Paste the JSON stored under the BASELINE key.

{
  "scannerVersion": "0.1.0",
  "commitSha": "0123456789abcdef0123456789abcdef01234567",
  "createdAt": "2026-10-23T12:00:12.000Z",
  "tools": [
    {
      "inventoryKey": "0123456789abcdef0123456789abcdef",
      "name": "get_weather",
      "description": "Fetch the current weather for a city.",
      "schemaHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
      "schemaNormalized": "{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}}}",
      "path": "src/index.ts"
    }
  ]
}

description, schemaHash, and schemaNormalized are null when the tool has no description or no static schema.

Output

A run writes three results. The run's output object links to each one.

• Findings dataset. One summary record, then one finding record per issue. Filter and export this table.
• OUTPUT key. The full report: coverage, summary, every finding, and the inventory. A failed run stores an error report here instead.
• BASELINE key. The inventory to pass as baseline on a later run. Present after a successful scan.

Example report

OUTPUT after a scan that found a missing tool description:

{
  "schemaVersion": "1.0.0",
  "scannerVersion": "0.1.0",
  "repository": {
    "fullName": "modelcontextprotocol/servers",
    "ref": "main",
    "commitSha": "0123456789abcdef0123456789abcdef01234567",
    "source": "github-public"
  },
  "status": "completed_with_findings",
  "startedAt": "2026-10-23T12:00:00.000Z",
  "completedAt": "2026-10-23T12:00:12.000Z",
  "coverage": {
    "filesConsidered": 42,
    "filesParsed": 40,
    "filesSkipped": 2,
    "toolDefinitionsDetected": 6,
    "unsupportedPatterns": 0,
    "limitReached": false
  },
  "summary": {
    "high": 1,
    "medium": 0,
    "low": 0,
    "info": 0,
    "confirmedDefects": 1,
    "heuristics": 0,
    "coverageNotes": 0
  },
  "findings": [
    {
      "id": "tool-004-a1b2c3d4e5f67890",
      "ruleId": "TOOL-004",
      "severity": "high",
      "confidence": "high",
      "kind": "deterministic",
      "toolName": "get_weather",
      "location": { "path": "src/index.ts", "startLine": 12, "endLine": 16 },
      "title": "Tool description is missing or empty",
      "evidence": { "message": "Tool \"get_weather\" has no description." },
      "recommendation": "Add a description that explains what the tool does and when to call it.",
      "references": ["https://modelcontextprotocol.io/specification/2025-11-25/server/tools"]
    }
  ]
}

status is completed, completed_with_findings, partial, unsupported, or failed. completed_with_findings means at least one high or medium deterministic finding. partial means a file cap stopped the scan early. unsupported means no recognized tool registration was found.

The same object includes inventory when the scan finishes. That object is also stored on its own under BASELINE.

Dataset records use the same finding object, wrapped for the table:

{ "type": "summary", "report": { "status": "completed_with_findings", "summary": { "high": 1 } } }

{ "type": "finding", "finding": { "ruleId": "TOOL-004", "severity": "high", "toolName": "get_weather", "title": "Tool description is missing or empty" } }

Example failure

A failed run stores this shape in OUTPUT and pushes one summary dataset record. error.code is INPUT_INVALID, REPO_NOT_FOUND, GITHUB_RATE_LIMIT, LIMIT_EXCEEDED, UNSUPPORTED_PATTERN, or INTERNAL_ERROR.

{
  "schemaVersion": "1.0.0",
  "scannerVersion": "0.1.0",
  "status": "failed",
  "error": {
    "code": "LIMIT_EXCEEDED",
    "message": "The scan exceeded a configured limit.",
    "diagnosticId": "a1b2c3d4e5f6"
  },
  "startedAt": "2026-10-23T12:00:00.000Z",
  "completedAt": "2026-10-23T12:00:04.000Z"
}

For AI agents

Call the Actor with repository set to owner/repo. The default checks are tool-definitions and input-schemas. Add baseline-diff and a baseline object only when you have a previous BASELINE record.

Read issues from the findings URL on the run output. Read status, coverage, and the inventory from the OUTPUT URL. Pass the BASELINE URL's JSON back as baseline on the next call.

Treat status: "failed" together with error.code as the failure contract. Count a defect when kind is deterministic and severity is high or medium. Info findings and kind: "coverage" are coverage or quality notes. summary.confirmedDefects counts deterministic findings. summary.coverageNotes counts coverage notes.

Rules

Info severity and coverage notes are separate from confirmed defects.

Tool definitions

Rule	Severity	What it reports
TOOL-001	high	Tool name is missing or empty.
TOOL-002	medium	Tool name is outside the MCP rules: 1–64 characters, only letters, digits, _, and -.
TOOL-003	high	Two tools in the same server share a name.
TOOL-004	high	Description is missing or blank.
TOOL-005	low	Description is under 10 characters. This is a heuristic.

Input schemas

Rule	Severity	What it reports
SCHEMA-001	high	inputSchema is missing or null.
SCHEMA-002	high	The schema object fails JSON Schema validation.
SCHEMA-003	medium	A required entry is missing from properties.
SCHEMA-004	info	The schema is a variable, call, or spread, so it cannot be checked statically.

Baseline diff

Rule	Severity	What it reports
BASELINE-001	high	A tool from the baseline is gone.
BASELINE-002	medium	The tool's input schema changed.
BASELINE-003	low	The tool's description changed.
BASELINE-004	info	A tool was added.

Coverage

Rule	Severity	What it reports
REPO-001	info	No recognized MCP tool registration in the files that were scanned.
SOURCE-001	info	A tool was detected and the parser could not attach a line number.

Supported MCP patterns

The scanner recognizes registrations from @modelcontextprotocol/sdk and static tool arrays in TypeScript or JavaScript.

McpServer.tool() positional form:

server.tool("get_weather", "Fetch weather", { city: z.string() }, handler);
server.tool("ping", handler);

McpServer.tool() object form:

server.tool({ name: "query", description: "Run query", inputSchema: { type: "object", properties: {} } });

Low-level Server.setRequestHandler():

server.setRequestHandler(ListToolsRequestSchema, async () => ({
  tools: [{ name: "my_tool", description: "Do the thing", inputSchema: { type: "object", properties: {} } }],
}));

A static tool array:

const tools: Tool[] = [
  { name: "calculator", description: "Math operations", inputSchema: { type: "object", properties: {} } },
];

Limitations

• The repository must be public on GitHub. The scan reads the git tree and file contents through the GitHub API.
• Source files must be TypeScript or JavaScript.
• Recognized registrations are the SDK patterns above and static tool arrays. Other frameworks show up as REPO-001 when nothing matches.
• A schema built at runtime is reported as SCHEMA-004. The scan records that the schema was dynamic.
• Defaults cap the scan at 5000 files, 500000 bytes per file, and 25000000 source bytes in total. Past a cap, coverage.limitReached is true and status is partial.

Pricing

A run has two event charges, and platform usage is billed as well. The Pricing tab on the Actor page lists the current rates.

• Actor start: $0.00005 when the run starts. One event covers up to 1 GB of memory, and each extra GB adds one event. The default memory is 1 GB, so a normal scan is one Actor start event. The compute cost of the first 5 seconds is waived.
• Result: $0.00001 for each row in the default dataset, which is $0.01 per 1,000 rows. A scan writes one summary row and one row per finding.
• Platform usage: compute for the rest of the run, at the rate of your Apify plan. Higher plans have a lower compute rate.

FAQ

Does this run the MCP server?

The Actor parses source text. It uses the Babel parser and never calls the tool handlers, eval, child_process, or npm install.

Which repos qualify?

Public GitHub repositories that contain a TypeScript or JavaScript MCP server. Pass owner/repo or the canonical GitHub URL. Private repositories are outside this Actor.

What is a baseline?

A baseline is the tool inventory from an earlier successful scan: tool name, description, schema hash, and file path, pinned to a commit SHA. Save the BASELINE key and pass that JSON as baseline with the baseline-diff check. The next run reports added, removed, and changed tools.

Why REPO-001?

REPO-001 means the scanned files had no registration this Actor recognizes. The repository can still contain an MCP server written in another language, or tools built only through dynamic code. Treat it as a coverage note.

How should an API or AI agent read the result?

Pass repository. Read the findings dataset URL from the run output for one record per issue. Read OUTPUT for status, summary, and error.code when the run failed. Use summary.confirmedDefects for deterministic findings. Leave info and coverage records out of a failure threshold. Schedule the same input, or call the run endpoint, when this check should repeat.