MCP-Atlas：一个大规模的工具使用能力基准测试，使用真实的MCP服务器
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
AI总结：
本文提出MCP-Atlas，一个用于评估工具使用能力的基准测试，基于真实MCP服务器，包含1000个由人类专家编写和验证的任务，覆盖36个真实MCP服务器和220种工具，通过任务级别的评估发现模型在工具调用和认知能力上的表现。
摘要：
模型上下文协议（MCP）正逐渐成为一种标准接口，通过该接口大型语言模型（LLM）代理可以发现并调用外部工具。然而，现有的MCP评估在三个方面存在不足：缺乏真实多步骤工作流和跨服务器编排、缺乏真实MCP服务器而非模拟器、以及缺乏结构化、可重复的声明级评分，与代理的冗长或风格无关。我们引入MCP-Atlas，一个用于测量工具使用能力的基准测试，针对生产MCP服务器。MCP-Atlas包含1000个自然语言任务，由人类专家编写和验证，涵盖36个真实MCP服务器和220种工具。提示不指定服务器、工具或参数，要求代理在语义上可能的干扰项中识别相关工具，并编排多步骤、跨服务器工作流。每个任务均使用声明级评分标准评分，最终答案根据工具输出中的原子事实声明评分。这种以答案为中心的评分允许有效的替代工具调用轨迹获得认可。我们将其与一个11类诊断分类法相结合，将工具调用失败与任务理解、综合、解析和停止的认知失败区分开来。在20个前沿模型（来自六个供应商）在匹配的任务级别条件下评估后，我们发现，在0.75声明覆盖率阈值下，通过率高达82.2%，并呈现出明显的三层性能结构。自动化诊断显示，63.3%的诊断失败是认知性的而非工具调用相关的。值得注意的是，一些高性能模型在成功执行工具后由于提前停止或综合错误而失败。我们发布了任务模式、容器化Harness、声明评估器和一个500任务的公共分割，同时保留500任务的私人分割以保持排行榜的完整性。代码在https://github.com/scaleapi/mcp-atlas。
英文摘要：
The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evaluations fall short along three key axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic MCP servers rather than mocks, and structured, reproducible claim-level scoring disentangled from agent verbosity or style. We introduce MCP-Atlas, a benchmark for measuring tool-use competency against production MCP servers. MCP-Atlas contains 1,000 natural-language tasks written and verified by human experts spanning 36 real MCP servers and 220 tools. Prompts do not specify servers, tools, or parameters, requiring agents to identify relevant tools among semantically plausible distractors and to compose multi-step, cross-server workflows. Each task is scored with a claim-level rubric, where final answers are scored against atomic factual claims grounded in tool outputs. This answer-centric scoring permits valid alternative tool-call trajectories to receive credit. We pair this with an 11-category diagnostic taxonomy that disentangles tool-call failures from cognitive failures in task understanding, synthesis, parsing, and stopping. Evaluating 20 frontier models from six providers under matched task-level conditions, we find pass rates up to 82.2% at a 0.75 claim coverage threshold and a clear three-tier performance structure. Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related. Notably, several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis. We release the task schema, containerized harness, claim evaluator, and a 500-task public split, while reserving a 500-task private split to preserve leaderboard integrity. The code is at https://github.com/scaleapi/mcp-atlas.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
