arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.18998cs.AIcs.CL
收藏

通用大语言模型代理的基准测试时间扩展

Evaluating Test-Time Scaling of General LLM Agents

Language Technologies Institute, School of Computer Science, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Meta

Xiaochuan Li, Ryan Ming, Pranav Setlur, Abhijay Paladugu, Andy Tang, Hao Kang, Shuai Shao, Rong Jin, Chenyan Xiong

AI总结:

研究提出General AgentBench基准测试,用于评估通用大语言模型代理在多个技能和工具上的表现,发现顺序和并行扩展方法在实际应用中均存在性能限制。

摘要:

大语言模型代理越来越多地被期望作为通用系统运作,能够解决开放性用户请求。虽然现有的基准测试专注于领域感知环境以开发专门的代理,但评估通用代理需要更现实的设置,使它们在统一环境中跨多个技能和工具进行操作。我们引入了General AgentBench,这是一个提供统一框架的基准测试,用于评估通用大语言模型代理在搜索、编程、推理和工具使用领域的能力。使用General AgentBench,我们系统地研究了在顺序扩展(迭代交互)和并行扩展(采样多个轨迹)下的测试时间扩展行为。对十个领先的大语言模型代理的评估显示,从领域特定评估转移到这种通用代理设置时性能显著下降。此外,我们发现由于两个根本限制:顺序扩展中的上下文天花板和并行扩展中的验证间隙,两种扩展方法在实践中都无法有效提升性能。代码可在https://github.com/cxcscmu/General-AgentBench公开获取。

英文摘要:

LLM agents are increasingly expected to operate as general-purpose systems that resolve real-world user requests, yet their dynamic scaling behavior in realistic environments remains poorly understood. In this paper, we systematically investigate two principal test-time scaling axes of LLM agents: sequential scaling through extended interaction and parallel scaling through trajectory sampling. We first introduce a realistic benchmark that provides one unified framework for evaluating LLM agents across search, coding, reasoning, and tool-use domains, more faithfully reflecting the heterogeneity of real-world deployments. Evaluating ten leading LLM agents reveals substantial performance degradation when transitioning from domain-specific evaluations to this realistic setting. Building on this foundation, we progressively scale test-time compute along fine-grained increments to characterize the performance upper bound. We find that neither scaling axis can consistently yield meaningful gains from additional test-time compute in realistic environments, a phenomenon we attribute to two fundamental limitations: the scaling plateau that bottlenecks sequential scaling and the verification gap that undermines parallel scaling. Code is publicly available at https://github.com/cxcscmu/General-AgentBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑