arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.18565cs.CLcs.AI
收藏

MINTEval: 评估长时间跨度智能体系统中的多目标干扰下的记忆

MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

UNC Chapel Hill(北卡罗来纳大学教堂山分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Hyunji Lee, Justin Chih-Yao Chen, Joykirat Singh, Zaid Khan, Elias Stengel-Eskin, Mohit Bansal

AI总结:

本文提出MINTEval基准,用于评估智能体在长时间跨度和多目标干扰下的记忆表现,通过长连接上下文、多领域和多类型问题来测试记忆增强代理的鲁棒性和泛化能力。

摘要:

现实中的智能体在长时间和不断演变的范围内运作,其中信息被不断更新并可能在记忆之间产生干扰,需要准确的回忆和对多份信息的聚合推理。然而,现有的基准主要关注静态、独立的回忆,无法捕捉这些动态的演变记忆之间的相互作用。在本文中,我们研究了当前的记忆增强代理在多样领域和问题类型中的长时间跨度、高干扰设置中的表现。我们引入MINTEval(长时间跨度记忆在干扰下的评估),该基准具有(1)长且高度互联的上下文,包含频繁更新的信息,从而产生显著的干扰;(2)多领域(状态跟踪、多轮对话、维基百科修订和GitHub提交),使能够评估领域泛化能力;(3)多类型问题,评估对干扰的鲁棒性,包括(i)单目标回忆任务,要求从长上下文中检索特定目标,以及(ii)多目标聚合任务,要求对多个相关信息片段进行推理。总体而言,MINTEval包含15.6k个问答对,覆盖平均138.8k个token的长时间跨度上下文,每个实例可扩展至1.8M个token。我们评估了7个代表性系统,包括 vanilla 长上下文 LLMs、RAG 和记忆增强代理框架。在所有系统中,我们观察到一致的低性能(平均27.9%准确率),尤其是在需要对多份证据进行聚合推理的问题上。我们的分析表明,性能主要受限于检索和记忆构建。此外,当前的记忆系统在面对被后续上下文修改或干扰的早期事实时,难以回忆和推理,准确性随着中间更新数量的增加而下降。

英文摘要:

Real-world agents operate over long and evolving horizons, where information is repeatedly updated and may interfere across memories, requiring accurate recall and aggregated reasoning over multiple pieces of information. However, existing benchmarks focus on static, independent recall and fail to capture these dynamic interactions between evolving memories. In this paper, we study how current memory-augmented agents perform in realistic, interference-heavy, long-horizon settings across diverse domains and question types. We introduce MINTEval (Long-Horizon Memory under INTerference Evaluation), a benchmark featuring (1) long, highly interconnected contexts with frequently updated information that induces substantial interference, (2) diverse domains (state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits), enabling evaluation of domain generalization, and (3) diverse question types that assess robustness to interference, including (i) single-target recall tasks requiring retrieval of a specific target from long contexts, and (ii) multi-target aggregation tasks requiring reasoning over multiple relevant pieces of information. Overall, MINTEval has 15.6k question-answering pairs over long-horizon contexts averaging 138.8k tokens and extending up to 1.8M tokens per instance. We evaluate 7 representative systems, including vanilla long-context LLMs, RAG, and memory-augmented agent frameworks. Across all systems, we observe consistently low performance (avg. 27.9% accuracy), especially on questions requiring aggregated reasoning over multiple pieces of evidence. Our analysis shows that performance is primarily limited by retrieval and memory construction. Furthermore, current memory systems struggle to recall and reason over earlier facts that are revised or interfered with by subsequent context, with accuracy degrading as the number of intervening updates increases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑