arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.04536cs.AI
收藏

通过参数化内存扩展自进化智能体

Scaling Self-Evolving Agents via Parametric Memory

Alibaba Group(阿里巴巴集团);Peking University(北京大学)

Tao Ren, Weiyao Luo, Hui Yang, Rongzhi Zhu, Xiang Huang, Yuchuan Wu, Bingxue Chou, Jieping Ye, Jiafeng Liang, Yongbin Li, Yijie Peng

AI总结:

提出TMEM框架,通过在线更新LoRA权重使智能体从经验中学习,从而在单轮交互中改变未来行为,并在多个基准上优于基于摘要和检索的方法。

摘要:

现有的内存增强型LLM智能体仅在提示空间中存储过去经验,作为文本摘要或检索段落,同时在整个运行过程中保持模型参数冻结。这类智能体可以\emph{查找}它们所见过的东西,但无法\emph{从中学习}:它们的策略不会因经验而改变,任何从上下文中丢弃的信息都会永久丢失。我们引入 exttt{TMEM},一个自进化的参数化内存框架,其中智能体不仅将历史压缩为显式内存,还通过轻量级在线更新将提炼的监督吸收到快速LoRA权重$Δ_t$中,从而在单个回合内真正改变其未来行为。我们将其形式化为具有快速权重运行动态的智能决策过程:动作从$π_{θ_0+Δ_t}$中采样,而提取动作产生监督,更新$Δ_t$以用于后续决策。这种观点使得提取策略可以直接通过RL优化:训练$θ_0$不仅改进了任务动作,还提高了用于在线LoRA适应的数据质量。我们进一步提出基于SVD的LoRA子空间初始化以加速在线收敛。在LoCoMo、LongMemEval-S、多目标搜索和CL-Bench上的实验表明, exttt{TMEM}在不同模型规模下始终优于基于摘要和基于检索的基线。

英文摘要:

Existing memory-augmented LLM agents store past experience exclusively in prompt space, as textual summaries or retrieved passages, while keeping model parameters frozen throughout a rollout. Such agents can \emph{look up} what they have seen but cannot \emph{learn from} it: their policy is unchanged by experience, and any information dropped from the context is permanently lost. We introduce \texttt{TMEM}, a self-evolving parametric memory framework in which the agent not only compresses history into explicit memory but also absorbs distilled supervision into fast LoRA weights $Δ_t$ via lightweight online updates, genuinely altering its future behavior within a single episode. We formalize this as an agentic decision process with fast-weight rollout dynamics: actions are sampled from $π_{θ_0+Δ_t}$, while extraction actions produce supervision that updates $Δ_t$ for subsequent decisions. This view makes the extraction policy directly optimizable by RL: training $θ_0$ improves not only task actions but also the quality of the data used for online LoRA adaptation. We further propose SVD-based initialization of the LoRA subspace to accelerate online convergence. Experiments on LoCoMo, LongMemEval-S, multi-objective search, and CL-Bench show that \texttt{TMEM} consistently outperforms summary-based and retrieval-based baselines across different model scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑