arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.23014cs.LGcs.CL
收藏

Mem-T: 长时间 horizon 记忆代理的奖励密集化

Mem-T: Densifying Rewards for Long-Horizon Memory Agents

Peking University(北京大学);Nanyang Technological University(南洋理工大学)

Yanwei Yue, Boci Peng, Xuanbo Fan, Jiaxin Guo, Qiankun Li, Yan Zhang

AI总结:

Mem-T通过引入树引导的强化学习框架,实现长时间 horizon 记忆管理的高效训练,显著提升性能并优化效率。

摘要:

记忆代理,通过内生管理记忆的处理、存储和检索,摆脱了预定义的记忆处理流程,因其自主性和适应性而受到越来越多的关注。然而,现有的训练范式仍然受限:代理往往在接收到稀疏且延迟的奖励之前需要经历长时间 horizon 的记忆操作序列,这阻碍了真正端到端优化记忆管理策略。为了解决这一限制,我们引入了Mem-T,一种自主的记忆代理,能够通过轻量级的分层记忆数据库进行动态更新和多轮检索,以处理流式输入。为了有效训练长时间 horizon 的记忆管理能力,我们进一步提出了MoT-GRPO,一种树引导的强化学习框架,通过记忆操作树反向传播和 hindsight credit assignment,将稀疏终端反馈转化为密集的逐步监督,从而实现记忆构建和检索的联合优化。广泛的实验表明,Mem-T在(1)高性能方面表现优异,比A-Mem和Mem0框架高出高达14.92%,并且(2)经济,运行在有利的准确率-效率帕累托前沿上,并且在不牺牲性能的情况下,将每查询的推理token减少约24.45%。

英文摘要:

Memory agents, which depart from predefined memory-processing pipelines by endogenously managing the processing, storage, and retrieval of memories, have garnered increasing attention for their autonomy and adaptability. However, existing training paradigms remain constrained: agents often traverse long-horizon sequences of memory operations before receiving sparse and delayed rewards, which hinders truly end-to-end optimization of memory management policies. To address this limitation, we introduce Mem-T, an autonomous memory agent that interfaces with a lightweight hierarchical memory database to perform dynamic updates and multi-turn retrieval over streaming inputs. To effectively train long-horizon memory management capabilities, we further propose MoT-GRPO, a tree-guided reinforcement learning framework that transforms sparse terminal feedback into dense, step-wise supervision via memory operation tree backpropagation and hindsight credit assignment, thereby enabling the joint optimization of memory construction and retrieval. Extensive experiments demonstrate that Mem-T is (1) high-performing, surpassing frameworks such as A-Mem and Mem0 by up to $14.92\%$, and (2) economical, operating on a favorable accuracy-efficiency Pareto frontier and reducing inference tokens per query by $\sim24.45\%$ relative to GAM without sacrificing performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑