An Innovative Leap in AI - Anthropic Develops Auditing Agents for Spotting Misalignment in AI Models
Companies have been gradually integrating the Model Context Protocol (MCP) into their systems, primarily assisting with identification and better use of agent tools. Salesforce researchers have gone a step further and found an additional application for MCP – aiding in the evaluation of AI agents themselves.
These scientists launched MCPEval, a new approach underpinned by an open-source toolkit built on the MCP system’s architecture, designed to scrutinise the performance of agents using tools. They noted existing evaluation methods were limited and often relied on fixed, predefined tasks, which fail to capture the real-world interactivity of agentic workflows.
The researchers explained that MCPEval transcends traditional success/failure metrics by systematically collecting task trajectories and protocol interaction data, resulting in unparalleled access to agent behaviour. Besides, by automating both task formation and validation, the derived high-quality trajectories enable quick agent model fine-tuning and continual improvement.