arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.20730cs.CL
收藏

AgentLongBench: 通过环境回放构建可控的长上下文代理基准

AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts

Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Soochow University(苏州大学)

Shicheng Fang, Yuxin Wang, Xiaoran Liu, Jiahao Lu, Chuanyuan Tan, Xinchi Chen, Yining Zheng, Xuanjing Huang, Xipeng Qiu

AI总结:

AgentLongBench通过模拟环境回放评估代理在长上下文任务中的表现,揭示代理在动态信息综合方面的不足。

摘要:

大型语言模型(LLMs)向自主代理的演进需要管理大量动态上下文。然而,当前基准仍然主要静态,依赖于被动检索任务,无法模拟代理与环境交互的复杂性,如非线性推理和迭代反馈。为此,我们引入AgentLongBench,通过基于横向思维谜题的模拟环境回放来评估代理。该框架在知识密集型和知识无关场景中生成严格交互轨迹。实验表明,尽管代理在静态检索方面表现良好,但在动态信息综合方面存在关键弱点。我们的分析表明,这种退化是由解决查询所需的最小token数驱动的。这一因素解释了为什么大规模工具响应固有的高信息密度比长回合对话中典型的内存碎片化更具挑战性。

英文摘要:

The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce \textbf{AgentLongBench}, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑