arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07962cs.AI
收藏

LOCA-bench: 评估在可控和极端上下文增长下的语言代理

LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth

Weihao Zeng, Yuzhen Huang, Junxian He

AI总结:

LOCA-bench通过可控和极端上下文增长评估语言代理的能力,提供长上下文代理的基准测试平台。

摘要:

大型语言模型(LLMs)越来越能够执行长时间的现实任务。然而,随着上下文量的增加,其可靠性往往下降,这种现象被称为

英文摘要:

Large language models (LLMs) are increasingly capable of carrying out long-running, real-world tasks. However, as the amount of context grows, their reliability often deteriorates, a phenomenon known as "context rot". Existing long-context benchmarks primarily focus on single-step settings that evaluate a model's ability to retrieve information from a long snippet. In realistic scenarios, however, LLMs often need to act as agents that explore environments, follow instructions and plans, extract useful information, and predict correct actions under a dynamically growing context. To assess language agents in such settings, we introduce LOCA-bench (a benchmark for LOng-Context Agents). Given a task prompt, LOCA-bench leverages automated and scalable control of environment states to regulate the agent's context length. This design enables LOCA-bench to extend the context length potentially to infinity in a controlled way while keeping the underlying task semantics fixed. LOCA-bench evaluates language agents as a combination of models and scaffolds, including various context management strategies. While agent performance generally degrades as the environment states grow more complex, advanced context management techniques can substantially improve the overall success rate. We open-source LOCA-bench to provide a platform for evaluating models and scaffolds in long-context, agentic scenarios: https://github.com/hkust-nlp/LOCA-bench

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑