从玩家到大师:通过基于记忆的强化学习增强LLM代理的测试时学习
From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory
AI总结:
提出Memopilot,一种通过多轮GRPO训练记忆更新过程来优化冻结LLM代理在测试时学习的方法,在多人博弈中显著提升Elo评分。
摘要:
大型语言模型(LLM)代理越来越多地部署在长期运行的环境中,在这些环境中,通过测试时的经验进行改进变得重要。一种常见的方法是在每次交互后更新显式记忆以指导未来的决策。然而,大多数现有方法依赖于手工设计的提示规则,这使得在多步时间跨度内难以使记忆更新与下游目标保持一致。我们提出Memopilot,一种即插即用的记忆副驾驶,它显式地训练记忆更新过程,以改进冻结的LLM在连续交互中的性能。我们将记忆更新公式化为一个多轮决策问题,并使用多轮GRPO进行端到端优化。我们的训练方案引入了(i)逐轮奖励信号和(ii)跨轨迹的上下文无关、逐轮优势估计,从而在多轮设置中实现更细粒度的信用分配和更稳定的训练。我们在两个测试平台上评估Memopilot:多轮石头剪刀布(RPS)和有限注德州扑克(LHE)。在这两种环境中,Memopilot显著改进了冻结玩家在测试时的学习,优于强基线,在两个游戏的Elo评分中均排名第一(LHE为1762,RPS为1590),并优于所有基线记忆方法和专有模型,包括DeepSeek-V3.2。
英文摘要:
Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock-Paper-Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。