MemQ：将Q学习整合到基于溯源DAG的自进化记忆代理中
MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs
AI总结：
MemQ通过在溯源DAG中传播信用权重，改进记忆Q值的评估，提升多步任务的泛化能力，尤其在产生深层相关溯源链的任务中表现突出。
摘要：
事件记忆使LLM代理能够积累和检索经验，但当前方法将每个记忆独立处理，即在不考虑通过记忆依赖链创建未来记忆的情况下评估检索质量。我们引入MemQ，将TD(λ) eligibility traces应用于记忆Q值，通过记录哪些记忆在创建每个新记忆时被检索的溯源DAG，将信用向后传播。信用权重随DAG深度d衰减为(γλ)^d，用结构接近代替时间距离。我们将该设定形式化为一个外源-上下文MDP，其分因过渡将外源任务流与内源记忆存储解耦。在六个基准测试中，涵盖操作系统交互、函数调用、代码生成、多模态推理、具身推理和专家级问答，MemQ在泛化评估和运行时学习中均取得最高成功率，最大增益出现在产生深层相关溯源链的多步任务（最高+5.7~pp），最小增益出现在单步分类任务（+0.77~pp）。我们进一步研究γ和λ如何与EC-MDP结构相互作用，为参数选择和未来研究提供指导。代码可在https://github.com/jwliao-ai/MemQ获取。
英文摘要：
Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retrieval quality in isolation without accounting for the dependency chains through which memories enable the creation of future memories. We introduce MemQ, which applies TD($λ$) eligibility traces to memory Q-values, propagating credit backward through a provenance DAG that records which memories were retrieved when each new memory was created. Credit weight decays as $(γλ)^d$ with DAG depth $d$, replacing temporal distance with structural proximity. We formalize the setting as an Exogenous-Context MDP, whose factored transition decouples the exogenous task stream from the endogenous memory store. Across six benchmarks, spanning OS interaction, function calling, code generation, multimodal reasoning, embodied reasoning, and expert-level QA, MemQ achieves the highest success rate on all six in generalization evaluation and runtime learning, with gains largest on multi-step tasks that produce deep and relevant provenance chains (up to +5.7~pp) and smallest on single-step classification (+0.77~pp) where single-step updates already suffice. We further study how $γ$ and $λ$ interact with the EC-MDP structure, providing principled guidance for parameter selection and future research. Code is available at https://github.com/jwliao-ai/MemQ.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
