arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.21147cs.AI
收藏

全能记忆:面向记忆智能体的过程奖励强化学习

All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents

Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Fudan University(复旦大学);The Hong Kong Polytechnic University(香港理工大学);Peng Cheng Laboratory(鹏城实验室)

Bingbing Wang, Jingjie Lin, Weiming Qiao, Jing Li, Ruifeng Xu

AI总结:

针对现有记忆方法缺乏共享奖励空间的问题,提出AiO-Mem模型,通过过程奖励强化学习优化记忆策略,在长上下文记忆基准上提升了答案准确率。

摘要:

大型语言模型越来越依赖外部记忆来回答基于长程历史的查询,例如多轮对话和长篇文档。然而,现有记忆方法通常分别优化检索和记忆构建,缺乏用于评估搜索、写入和停止决策的共享奖励空间。为解决该问题,我们提出全能记忆(All-in-One Memory,AiO-Mem),将这些操作组织为共享证据状态上的转移,并在统一奖励空间中直接评估每个动作的语义质量。AiO-Mem将异构历史编译为可溯源的多视图事件,为检索和记忆写入提供共享基础。随后,一个独立的轨迹条件过程奖励模型通过基于证据的监督微调以及轨迹级别的DPO训练,为收集到的轨迹分配过程分数。这些分数被聚合为动作平衡的轨迹奖励,并由过程对齐的DAPO(Process-Aligned DAPO,PA-DAPO)用于优化记忆策略。在长上下文记忆基准测试中,AiO-Mem相比静态记忆系统和强化学习训练的记忆智能体,始终提升了答案准确率。

英文摘要:

Large language models increasingly rely on external memory to answer queries grounded in long-horizon histories such as multi-session dialogues and lengthy documents. However, existing memory methods often optimize retrieval and memory construction separately, without a shared reward space for evaluating search, writing, and stopping decisions. To address this, we propose All-in-One Memory (AiO-Mem), which organizes these operations as transitions over a shared evidence state and directly evaluates the semantic quality of each action in a unified reward space. AiO-Mem compiles heterogeneous histories into source-traceable multi-view events, providing a shared substrate for retrieval and memory writing. A unified memory policy actor then rolls out complete interaction trajectories over this shared evidence state. A separate trajectory-conditioned process reward model is trained through evidence-grounded supervised fine-tuning and trajectory-level DPO to assign process scores to the collected trajectories. These scores are aggregated into an action-balanced trajectory reward and used by Process-Aligned DAPO (PA-DAPO) to optimize the memory policy. Across long-context memory benchmarks, AiO-Mem consistently improves answer accuracy over both static memory systems and RL-trained memory agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑