ComplexMCP: 评估LLM代理在动态、相互依赖和大规模工具沙箱中的表现
ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox
AI总结：
本文提出ComplexMCP基准，用于评估LLM代理在动态、相互依赖和大规模工具环境中的性能，揭示了现有模型在复杂任务中的不足，指出三个关键瓶颈：工具检索饱和、过度自信和战略投降倾向。
摘要：
当前LLM代理擅长调用孤立API，但在商业软件自动化最后一公里方面表现不佳。在现实场景中，工具并非独立，而是原子性、相互依赖且易受环境噪声影响。我们引入ComplexMCP，一个基于Model Context Protocol（MCP）设计的基准，提供超过300个经过严格测试的工具，来源于7个状态沙箱，涵盖办公套件到金融系统。与现有数据集不同，我们的基准采用种子驱动架构模拟动态环境状态和不可预测的API故障，确保评估的确定性与多样性。我们评估了各种LLM在全上下文和RAG范式下的表现，揭示了显著的性能差距：即使顶级模型也难以超过60%的成功率，远低于人类90%的表现。细粒度轨迹分析识别出三个根本瓶颈：（1）工具检索饱和；（2）过度自信，即代理跳过必要的环境验证；（3）战略投降倾向，即倾向于合理化失败而非追求恢复。这些发现凸显了当前代理在相互依赖工作流中的不足，将ComplexMCP定位为下一代鲁棒自主系统的关键测试平台。
英文摘要：
Current LLM agents are proficient at calling isolated APIs but struggle with the "last mile" of commercial software automation. In real-world scenarios, tools are not independent; they are atomic, interdependent, and prone to environmental noise. We introduce $\textbf{ComplexMCP}$, a benchmark designed to evaluate agents in these rigorous conditions. Built on the Model Context Protocol (MCP), $\textbf{ComplexMCP}$ provides over 300 meticulously tested tools derived from 7 stateful sandboxes, ranging from office suites to financial systems. Unlike existing datasets, our benchmark utilizes a seed-driven architecture to simulate dynamic environment states and unpredictable API failures, ensuring a deterministic yet diverse evaluation. We evaluate various LLMs across full-context and RAG paradigms, revealing a stark performance gap: even top-tier models fail to exceed a 60% success rate, far trailing human performance 90%. Granular trajectory analysis identifies three fundamental bottlenecks: (1) $\textbf{tool retrieval saturation}$ as action spaces scale; (2) $\textbf{over-confidence}$, where agents skip essential environment verifications; and (3) $\textbf{strategic defeatism}$, a tendency to rationalize failure rather than pursuing recovery. These findings underscore the insufficiency of current agents for interdependent workflows, positioning $\textbf{ComplexMCP}$ as a critical testbed for the next generation of resilient autonomous systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
