arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.01353cs.AIcs.CL
收藏

MEMTRACK:评估多平台动态智能体环境中的长期记忆与状态追踪

MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments

Patronus AI

Darshan Deshpande, Varun Gangal, Hersh Mehta, Anand Kannappan, Rebecca Qian, Peng Wang

AI总结:

本研究推出多平台动态智能体环境记忆基准MEMTRACK,模拟真实企业工作流测试记忆能力,构建真实场景数据集并提出多维度指标,实验发现前沿模型表现不佳,为相关评估研究提供可扩展框架。

摘要:

近期关于上下文与记忆基准测试的研究主要聚焦于对话场景,但评估动态企业环境中的记忆能力对其有效应用至关重要。我们推出MEMTRACK,这是一款专为评估多平台智能体环境中的长期记忆与状态追踪能力设计的基准测试。\nMEMTRACK通过整合Slack、Linear、Git等多个沟通与生产力平台的异步事件,模拟真实的组织工作流程。每个基准测试实例提供按时间顺序交错的跨平台时间线,包含噪声信息、冲突信息、交叉引用信息,还可能涉及代码库/文件系统的理解与探索任务。因此,该基准测试可检验记忆的获取、选择与冲突解决等能力。\n我们通过专家手动设计与可扩展的基于智能体的合成两种方式构建MEMTRACK数据集,生成的场景均基于真实软件开发流程,具有生态有效性。我们还提出了正确性、效率与冗余度三项相关指标,用以衡量记忆机制的有效性,而非仅关注简单的问答性能。\n在前沿大语言模型(LLM)与记忆后端上开展的实验显示,现有系统在长周期记忆利用、跨平台依赖处理与矛盾解决方面存在挑战。值得注意的是,表现最佳的GPT-5模型在MEMTRACK上的正确性得分仅为60%。\n本研究提供了一个可扩展框架,可推动记忆增强智能体的评估研究突破现有对话场景的局限,为复杂组织场景下的多智能体、多平台记忆基准测试奠定基础。

英文摘要:

Recent works on context and memory benchmarking have primarily focused on conversational instances but the need for evaluating memory in dynamic enterprise environments is crucial for its effective application. We introduce MEMTRACK, a benchmark designed to evaluate long-term memory and state tracking in multi-platform agent environments. MEMTRACK models realistic organizational workflows by integrating asynchronous events across multiple communication and productivity platforms such as Slack, Linear and Git. Each benchmark instance provides a chronologically platform-interleaved timeline, with noisy, conflicting, cross-referring information as well as potential codebase/file-system comprehension and exploration. Consequently, our benchmark tests memory capabilities such as acquistion, selection and conflict resolution. We curate the MEMTRACK dataset through both manual expert driven design and scalable agent based synthesis, generating ecologically valid scenarios grounded in real world software development processes. We introduce pertinent metrics for Correctness, Efficiency, and Redundancy that capture the effectiveness of memory mechanisms beyond simple QA performance. Experiments across SoTA LLMs and memory backends reveal challenges in utilizing memory across long horizons, handling cross-platform dependencies, and resolving contradictions. Notably, the best performing GPT-5 model only achieves a 60\% Correctness score on MEMTRACK. This work provides an extensible framework for advancing evaluation research for memory-augmented agents, beyond existing focus on conversational setups, and sets the stage for multi-agent, multi-platform memory benchmarking in complex organizational settings

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑