arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.08435cs.CL
收藏

细粒度反馈对齐的长期记忆管理:Fine-Mem

Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management

Harbin Institute of Technology(哈尔滨工业大学);Meituan(美团);Peng Cheng Laboratory(鹏城实验室);The University of Hong Kong(香港大学)

Weitao Ma, Xiaocheng Feng, Lei Huang, Xiachong Feng, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, Bing Qin

AI总结:

Fine-Mem通过细粒度反馈对齐提升长周期内存管理,通过块级奖励和证据锚定奖励分配优化策略,实现更高效的内存操作与长期效用对齐。

摘要:

有效的内存管理对于大型语言模型代理在长周期任务中导航至关重要。近期研究探索了使用强化学习开发专门的内存管理代理。然而,现有方法依赖最终任务表现作为主要奖励,导致严重的奖励稀疏性和无效的信用分配,无法为个体内存操作提供充分指导。为此,我们提出了Fine-Mem,一个统一的框架,用于细粒度反馈对齐。首先,我们引入了块级步骤奖励,通过辅助的块特定问答任务提供即时的步骤级监督。其次,我们设计了证据锚定奖励分配,通过将信用锚定到关键内存操作来重新分配全局奖励,基于在推理中使用的特定内存项目作为证据。这些组件共同实现了稳定的策略优化,并将局部内存操作与内存的长期效用对齐。在Memalpha和MemoryAgentBench上的实验表明,Fine-Mem在各种子任务中一致优于强基线,实现了更高的成功率。进一步分析揭示了其在不同模型配置和主干网络上的适应性和强大的泛化能力。

英文摘要:

Effective memory management is essential for large language model agents to navigate long-horizon tasks. Recent research has explored using Reinforcement Learning to develop specialized memory manager agents. However, existing approaches rely on final task performance as the primary reward, which results in severe reward sparsity and ineffective credit assignment, providing insufficient guidance for individual memory operations. To this end, we propose Fine-Mem, a unified framework designed for fine-grained feedback alignment. First, we introduce a Chunk-level Step Reward to provide immediate step-level supervision via auxiliary chunk-specific question answering tasks. Second, we devise Evidence-Anchored Reward Attribution to redistribute global rewards by anchoring credit to key memory operations, based on the specific memory items utilized as evidence in reasoning. Together, these components enable stable policy optimization and align local memory operations with the long-term utility of memory. Experiments on Memalpha and MemoryAgentBench demonstrate that Fine-Mem consistently outperforms strong baselines, achieving superior success rates across various sub-tasks. Further analysis reveals its adaptability and strong generalization capabilities across diverse model configurations and backbones.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑