MemBuilder: 通过具有属性的密集奖励强化LLMs以进行长期记忆构建
MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards
AI总结:
MemBuilder通过具有属性的密集奖励强化LLMs,解决长期对话中保持一致性的挑战,提升多维记忆构建能力,实验表明其在长期对话基准测试中表现优异。
摘要:
维持长期对话中的一致性仍然是LLMs面临的基本挑战,因为标准检索机制往往无法捕捉历史状态的时序演变。虽然记忆增强框架提供了一个结构化的替代方案,但当前系统依赖于闭源模型的静态提示或遭受稀疏奖励训练范式的无效影响。我们引入MemBuilder,一种强化学习框架,训练模型通过具有属性的密集奖励来协调多维记忆构建。MemBuilder解决了两个关键挑战:(1) 稀疏轨迹级奖励:我们采用合成会话级问题生成,为扩展轨迹提供密集的中间奖励;(2) 多维记忆归因:我们引入了贡献感知的梯度加权,根据每个组件的下游影响调整策略更新。实验结果表明,MemBuilder使400亿参数模型在与最新闭源基线相比时表现更优,并在长期对话基准测试中表现出强大的泛化能力。
英文摘要:
Maintaining consistency in long-term dialogues remains a fundamental challenge for LLMs, as standard retrieval mechanisms often fail to capture the temporal evolution of historical states. While memory-augmented frameworks offer a structured alternative, current systems rely on static prompting of closed-source models or suffer from ineffective training paradigms with sparse rewards. We introduce MemBuilder, a reinforcement learning framework that trains models to orchestrate multi-dimensional memory construction with attributed dense rewards. MemBuilder addresses two key challenges: (1) Sparse Trajectory-Level Rewards: we employ synthetic session-level question generation to provide dense intermediate rewards across extended trajectories; and (2) Multi-Dimensional Memory Attribution: we introduce contribution-aware gradient weighting that scales policy updates based on each component's downstream impact. Experimental results show that MemBuilder enables a 4B-parameter model to outperform state-of-the-art closed-source baselines, exhibiting strong generalization across long-term dialogue benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。