arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.03329cs.AI
收藏

InfoMem: 基于答案条件信息增益训练长上下文记忆智能体

InfoMem: Training Long-Context Memory Agents with Answer-Conditioned Information Gain

Tongji University(同济大学);Shanghai Innovation Institute(上海创新研究院);Shanghai AI Laboratory(上海人工智能实验室)

Tiancheng Han, Yong Li, Wuzhou Yu, Qiaosheng Zhang, Wenqi Shao

AI总结:

提出InfoMem奖励机制,通过评估最终记忆对真实答案的每token对数似然增益,训练分块记忆智能体以提升长上下文任务性能。

摘要:

长上下文任务要求LLM从大上下文中识别并保留与答案相关的信息。分块记忆智能体通过顺序读取文档块、更新紧凑记忆并从累积记忆中生成最终答案来解决这一问题。然而,现有的基于RL的分块智能体要么依赖稀疏的最终答案奖励,要么使用词汇中间奖励来指导记忆和检索动作。这些信号监督任务成功或局部重叠,但不直接评估最终记忆是否支持真实答案。我们提出InfoMem,一种用于训练分块记忆智能体的奖励机制,该机制使用答案条件信息评估最终记忆的效用。InfoMem衡量最终记忆增加模型对真实答案的每token对数似然的程度。为了稳定RL优化,InfoMem仅对成功轨迹应用此信号,并在奖励组合前对其进行归一化。在相同的GRPO框架和训练预算下,InfoMem在长上下文记忆智能体性能上优于可比的记忆智能体RL基线。分析表明,有效的最终记忆奖励应作用于成功轨迹,在奖励组合前归一化,并基于答案而非查询进行条件化。我们的代码可从此https URL获取。

英文摘要:

Long-context tasks require LLMs to identify and preserve answer-relevant information from large contexts. Chunk-wise memory agents address this issue by sequentially reading document chunks, updating a compact memory, and generating the final answer from the accumulated memory. However, existing RL-based chunk-wise agents either rely on sparse final-answer rewards or use lexical intermediate rewards for memory and retrieval actions. These signals supervise task success or local overlap, but do not directly evaluate whether the final memory supports the ground-truth answer. We propose InfoMem, a reward mechanism for training chunk-wise memory agents that evaluates final-memory utility using answer-conditioned information. InfoMem measures how much the final memory increases the model's per-token log-likelihood of the ground-truth answer. To stabilize RL optimization, InfoMem applies this signal only to successful trajectories and normalizes it before reward composition. Under the same GRPO framework and training budget, InfoMem improves long-context memory-agent performance over comparable memory-agent RL baselines. Analyses show that effective final-memory rewards should operate on successful trajectories, be normalized before reward composition, and be conditioned on the answer rather than the query. Our code is available at https://github.com/GenSouKa1/InfoMem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑