AI Agent Tool Selection Optimizer

Runtime evidence for smaller, cheaper, more reliable MCP and AI-agent toolsets.

This Actor analyzes observed tool usage telemetry, not just tool names or descriptions. It combines call volume, reliability, latency, cost, and declared task coverage to rank tools, identify prune candidates, preserve unique capabilities, and generate a deterministic routing policy.

Use cases

• MCP servers exposing many tools to an agent
• agent platforms with repeated wrong-tool calls
• AI FinOps teams reducing tool/API spend
• engineering teams comparing overlapping functions or connectors
• CI or weekly governance of production agent toolsets
• teams trying to reduce tool-schema/context overhead without blindly deleting capabilities

Why use this Actor

Static tool-catalog pruning can tell you which tools look similar. This Actor works from what your production or test agents actually did. It can use:

• observed call counts
• successful or failed calls
• average and p95 latency
• average cost per call
• task types served by each tool
• an optional estimate of context tokens consumed per exposed tool

Key features

• Runtime scoring from usage, reliability, latency, and cost telemetry
• Task-aware routing policy that preserves unique capabilities
• Explicit KEEP, BENCHMARK, and PRUNE_CANDIDATE recommendations
• Deterministic analysis with no external LLM or paid API dependency

The output preserves the best observed tool for each declared task type even when the requested toolset cap is smaller. That reduces the risk of optimizing cost by accidentally removing a unique capability.

No external LLM or paid API is used. Analysis is deterministic and low-compute.

Input

tool_usage_logs is required. Each row needs tool_id and usage_count. Add the optional telemetry fields you have; richer telemetry increases the usefulness of the recommendation.

Example

{
  "tool_usage_logs": [
    {
      "tool_id": "internal_docs_search",
      "usage_count": 1200,
      "success_count": 1176,
      "avg_latency_ms": 180,
      "p95_latency_ms": 360,
      "avg_cost_usd": 0.0015,
      "task_types": ["knowledge_search"]
    },
    {
      "tool_id": "web_search",
      "usage_count": 420,
      "success_count": 382,
      "avg_latency_ms": 820,
      "p95_latency_ms": 1450,
      "avg_cost_usd": 0.009,
      "task_types": ["knowledge_search", "current_web_research"]
    }
  ],
  "optimization_mode": "balanced",
  "max_recommended_tools": 12,
  "context_tokens_per_tool": 180
}

Optimization modes

• balanced — reliability, cost, latency, and observed usage
• cost — emphasize cheaper successful calls
• reliability — emphasize successful calls
• latency — emphasize faster tools
• compact — emphasize a smaller high-usage set while preserving task coverage

Output

A run emits one default-dataset item containing the complete audit:

• ranked recommendations for every analyzed tool
• KEEP, BENCHMARK, or PRUNE_CANDIDATE action
• utility score and observed telemetry
• task-specific preferred-tool routing policy
• reliability risk list
• observed spend attributable to prune candidates
• potential tool-context reduction
• evidence confidence level

The same result is also written to the OUTPUT key-value-store record.

Limitations

Interpreting prune candidates

A prune candidate is not an instruction to remove a production capability immediately. It means the observed telemetry does not justify keeping that tool in the default exposed set relative to the alternatives and declared task coverage. Replay representative tasks before removing or disabling it.

Data provenance and privacy

The Actor uses only telemetry supplied in the Actor input. It does not contact external model providers, MCP servers, websites, or paid APIs. Do not submit secrets or raw user content when aggregated operational metrics are sufficient.

Pricing

One successfully emitted audit equals one billable default-dataset item. The factory profit guard validates real Apify run cost before initial Store pricing is activated.

Search terms

AI agent tool selection, MCP tool optimization, MCP routing audit, AI agent FinOps, reduce wrong tool calls, tool reliability benchmark, agent tool cost optimization, MCP context reduction, agent routing policy.

Commercial intelligence layer

The Gen2 analysis can compare against a previous audit, translate tool pruning and context reduction into projected monthly and annual savings, produce a numeric confidence score, and return an explicit GO / WARN / BLOCK decision. Existing runtime ranking and routing fields remain unchanged for backward compatibility.