模型上下文协议(MCP)工具描述存在坏味道!通过增强MCP工具描述提升AI智能体效率
Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions
AI总结:
本文通过实证研究103个MCP服务器中的856个工具描述,发现97.1%的描述存在坏味道,并提出增强描述方法,在提升任务成功率(中位数+5.85%)的同时增加了执行步骤(+67.46%),进一步通过组件消融实验找到兼顾性能与效率的紧凑描述组合。
摘要:
模型上下文协议(MCP)引入了一个标准规范,定义了基于基础模型(FM)的智能体应如何通过调用工具与外部系统交互。然而,为了理解工具的目的和功能,FM依赖自然语言工具描述,这使得这些描述成为引导FM为给定(子)任务选择最优工具并向工具传递正确参数的关键组件。尽管这些描述中的缺陷或坏味道可能误导基于FM的智能体,但其在MCP生态系统中的普遍性和后果仍不清楚。因此,我们实证研究了分布在103个MCP服务器中的856个工具,评估其描述质量及其对智能体性能的影响。我们从文献中识别出工具描述的六个组件,利用这些组件制定评分标准,然后基于该标准形式化工具描述坏味道。通过基于FM的扫描器操作该标准,我们发现97.1%的分析工具描述包含至少一个坏味道,其中56%未能清晰陈述其目的。虽然为所有组件增强这些描述使任务成功率中位数提高5.85个百分点,部分目标完成率提高15.12%,但同时也使执行步骤增加了67.46%,并在16.67%的情况下导致性能倒退。这些结果表明实现性能提升并非易事;虽然执行成本可以作为权衡,但执行上下文也会产生影响。此外,组件消融实验表明,不同组件组合的紧凑变体通常能在保持行为可靠性的同时减少不必要的令牌开销,从而更有效地利用FM上下文窗口并降低执行成本。
英文摘要:
The Model Context Protocol (MCP) introduces a standard specification that defines how Foundation Model (FM)-based agents should interact with external systems by invoking tools. However, to understand a tool's purpose and features, FMs rely on natural-language tool descriptions, making these descriptions a critical component in guiding FMs to select the optimal tool for a given (sub)task and to pass the right arguments to the tool. While defects or smells in these descriptions can misguide FM-based agents, their prevalence and consequences in the MCP ecosystem remain unclear. Hence, we examine 856 tools spread across 103 MCP servers empirically, assess their description quality, and their impact on agent performance. We identify six components of tool descriptions from the literature, develop a scoring rubric utilizing these components, and then formalize tool description smells based on this rubric. By operationalizing this rubric through an FM-based scanner, we find that 97.1% of the analyzed tool descriptions contain at least one smell, with 56% failing to state their purpose clearly. While augmenting these descriptions for all components improves task success rates by a median of 5.85 percentage points and improves partial goal completion by 15.12%, it also increases the number of execution steps by 67.46% and regresses performance in 16.67% of cases. These results indicate that achieving performance gains is not straightforward; while execution cost can act as a trade-off, execution context can also impact. Furthermore, component ablations show that compact variants of different component combinations often preserve behavioral reliability while reducing unnecessary token overhead, enabling more efficient use of the FM context window and lower execution costs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。