OpenAI Agent MCP Tester

It is important to ensure that your MCP server is functioning correctly and that all tools are available and working as expected. This Actor automates the manual testing process using OpenAI agents and provides a detailed report on the status of each tool.

ℹ️ This project uses the OpenAI Agents Python SDK for agent orchestration and MCP tool testing.

Since MCP servers are often used in agentic-based applications, it is essential to test the MCP server using an agentic approach with intelligent test planning and execution.

🔥 Features

• Connects to multiple MCP servers via URL and tests all available tools
• Uses OpenAI agents framework with GPT-4.1 Mini for intelligent test planning and execution
• Two-agent orchestrator pattern: planner agent generates test scenarios, executor agent runs tests
• Generates comprehensive test reports with pass/fail status and detailed explanations
• Supports custom headers for authentication
• HTTP streaming connection for reliable remote server testing
• Token-based pricing model using Pay per Event
• Runs on Apify platform

📊 Output data

The Actor provides comprehensive test results through two output formats:

Dataset results

Individual test records for each tool tested. Each record contains:

Field	Type	Description
mcp_url	String	URL of the MCP server that was tested
tool_name	String	Name of the MCP tool that was tested
parameters	Object	Parameters used for the test
query	String	Input query used for testing
output	String	Actual output from the tool
passed	Boolean	Whether the tool test passed (true/false)
explanation	String	Detailed explanation of the test result

Key-value store results

Overall test summaries for each MCP server with aggregated statistics:

Field	Type	Description
mcp_url	String	URL of the MCP server that was tested
summary	Object	Summary statistics for the MCP server
summary.total_tests	Integer	Total number of tests executed
summary.passed_tests	Integer	Number of tests that passed
summary.failed_tests	Integer	Number of tests that failed
summary.all_passed	Boolean	Whether all tests passed for this MCP server
test_results	Array	Detailed results for each individual test (same structure as dataset records)

🚀 Usage

1. Navigate to the Actor page
2. Provide the MCP server URLs (supports multiple servers)
3. Add authentication headers for the MCP servers (if required)
4. Start the test
5. Review the generated comprehensive report

💰 Pricing

You will be charged based on LLM token usage. For a simple MCP server with a few tools, the cost should be under $0.05 per test run.

ℹ️ Notice:
This Actor internally uses the Apify Openrouter Actor to call an LLM. You will be billed for LLM usage through this Actor.

💾 Input

The Actor accepts the following input parameters:

{
  "mcpUrls": [
    "https://your-mcp-server-1.com/mcp",
    "https://your-mcp-server-2.com/mcp"
  ],
  "headers": {
    "Authorization": "Bearer your-token",
    "X-Custom-Header": "custom-value"
  }
}

Parameters:

• mcpUrls (required): Array of MCP server endpoint URLs
• headers (optional): HTTP headers for authentication or configuration

Automatic Apify.com Authorization:

If any MCP URL contains apify.com, the Actor will automatically set the Authorization header to Bearer <APIFY_TOKEN> using the value from the APIFY_TOKEN environment variable. This is handled internally as follows:

for i, mcp_url in enumerate(mcp_urls, 1):
    if 'apify.com' in mcp_url or 'apify.actor' in mcp_url:
        headers['Authorization'] = f'Bearer {os.getenv("APIFY_TOKEN")}'

This means you do not need to manually provide the Authorization header for Apify MCP servers if the APIFY_TOKEN environment variable is set.

🔢 Output

The Actor provides two types of output:

Test results (Dataset)

Individual test records for each tool tested. Each record is a separate JSON object:

[
  {
    "mcp_url": "https://mcp.apify.com",
    "tool_name": "search-actors",
    "parameters": {"search": "test"},
    "query": "Search for actors with keyword 'test'",
    "output": "Found 5 actors matching 'test'",
    "passed": true,
    "explanation": "Successfully searched for actors and received valid results"
  },
  {
    "mcp_url": "https://mcp.apify.com",
    "tool_name": "get-actor-details",
    "parameters": {"actor": "apify/proxy-test"},
    "query": "Get details for actor 'apify/proxy-test'",
    "output": "Actor details retrieved successfully",
    "passed": true,
    "explanation": "Successfully retrieved detailed information about the actor"
  }
]

Test summaries (Key-value store)

Overall test summaries for each MCP server with aggregated statistics:

{
  "mcp_url": "https://mcp.apify.com",
  "summary": {
    "total_tests": 5,
    "passed_tests": 4,
    "failed_tests": 1,
    "all_passed": false
  },
  "test_results": [
    {
      "tool_name": "search-actors",
      "parameters": {"search": "test"},
      "query": "Search for actors with keyword 'test'",
      "output": "Found 5 actors matching 'test'",
      "passed": true,
      "explanation": "Successfully searched for actors and received valid results",
      "mcp_url": "https://mcp.apify.com"
    }
  ]
}

🔧 Configuration

Custom headers

For MCP servers requiring authentication:

{
  "mcpUrls": ["https://secure-mcp-server.com/mcp"],
  "headers": {
    "Authorization": "Bearer your-secure-token",
    "X-API-Key": "your-api-key"
  }
}

🏗️ Architecture

Two-agent orchestrator pattern

The tester uses two-agent architecture that was designed based on extensive testing of different approaches:

Why two agents?

After experimenting with various agent patterns, we found that using a single agent for both planning and execution often fails - the agent gets overwhelmed and struggles to execute all planned tests systematically. Similarly, using a three-agent orchestration pattern (planner, executor, and judge) leads to overly complex interactions and longer run times.

The two-agent approach (planner + executor) proved to be the most effective solution because it provides:

• Clear separation of concerns: One agent focuses solely on test planning, the other on execution
• Better control over test cases: Each agent can be optimized for its specific role
• Faster debugging: When something goes wrong, it's easy to identify whether the issue is in planning or execution
• Reliable test coverage: The planner agent can systematically analyze all available tools without getting distracted by execution details

This pattern aligns with the OpenAI Agents SDK multi-agent orchestration principles, which recommend separating complex tasks into specialized agents for better control and performance.

The tester uses this proven architecture:

1. Planner agent (agent-plan-tests):

   • Analyzes available MCP tools
   • Generates comprehensive test scenarios
   • Focuses on happy paths and typical use cases
   • Creates structured test cases with expected outputs

2. Executor agent (agent-exec-test):

   • Executes test scenarios by calling MCP tools
   • Evaluates results and determines pass/fail status
   • Provides detailed explanations for each test result
   • Handles retries and error recovery

📚 Resources

This Actor is open source and available on GitHub.

• OpenAI Agents Python SDK documentation
• OpenAI Agents multi-agent orchestration guide
• Building Agents with OpenAI
• Apify Actor OpenAI Agents template
• How to Build an AI Agent with OpenAI Agents
• How to use MCP with Apify Actors
• Webinar: Building and monetizing MCP servers on Apify
• Apify MCP server documentation
• Apify MCP server configuration