TheMCPCompany:创建通用代理的专用工具
TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
AI总结:
TheMCPCompany通过评估工具调用代理在现实世界服务交互中的表现,揭示了当前模型在复杂任务中的挑战。
摘要:
自从引入模型上下文协议(MCP)以来,大型语言模型(LLMs)可用的工具数量显著增加。这些任务专用工具集为通用工具如网络浏览器提供了替代方案,同时比图形用户界面(GUI)更容易开发和维护。然而,当前通用代理主要依赖网络浏览器与环境交互。在此,我们引入TheMCPCompany,一个用于评估在涉及与各种现实世界服务交互的任务中调用工具的代理的基准。我们使用这些服务的REST API创建MCP服务器,其中包括超过18,000个工具。我们还为每个任务提供人工标注的地面真实工具。在我们的实验中,我们使用地面真实工具来展示调用代理在提高性能和降低成本方面的潜力,假设工具检索完美。接下来,我们通过工具检索探索代理性能,以研究基于工具的代理在现实中的实用性。尽管所有具有工具检索的模型在性能上与基于浏览器的代理相似或更好,但较小的模型无法通过检索充分利用可用的工具。另一方面,GPT-5在使用工具检索时的表现与使用地面真实工具时的表现非常接近。总体而言,我们的工作表明,最先进的推理模型在简单的环境中能够有效地发现工具,但在复杂的企业环境中则严重挣扎。TheMCPCompany揭示了在数万种工具中导航并以非平凡方式组合它们以解决复杂问题仍然是当前模型的挑战,需要更好的推理和更好的检索模型。
英文摘要:
Since the introduction of the Model Context Protocol (MCP), the number of available tools for Large Language Models (LLMs) has increased significantly. These task-specific tool sets offer an alternative to general-purpose tools such as web browsers, while being easier to develop and maintain than GUIs. However, current general-purpose agents predominantly rely on web browsers for interacting with the environment. Here, we introduce TheMCPCompany, a benchmark for evaluating tool-calling agents on tasks that involve interacting with various real-world services. We use the REST APIs of these services to create MCP servers, which include over 18,000 tools. We also provide manually annotated ground-truth tools for each task. In our experiments, we use the ground truth tools to show the potential of tool-calling agents for both improving performance and reducing costs assuming perfect tool retrieval. Next, we explore agent performance using tool retrieval to study the real-world practicality of tool-based agents. While all models with tool retrieval perform similarly or better than browser-based agents, smaller models cannot take full advantage of the available tools through retrieval. On the other hand, GPT-5's performance with tool retrieval is very close to its performance with ground-truth tools. Overall, our work shows that the most advanced reasoning models are effective at discovering tools in simpler environments, but seriously struggle with navigating complex enterprise environments. TheMCPCompany reveals that navigating tens of thousands of tools and combining them in non-trivial ways to solve complex problems is still a challenging task for current models and requires both better reasoning and better retrieval models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。