通过回顾轨迹重写实现LM代理的样本高效在线学习
Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting
AI总结：
ECHO通过回顾轨迹重写提升LM代理的样本效率，有效利用过去经验以更快适应新环境。
摘要：
语言模型（LM）代理在新环境中部署时，从序列交互中学习往往表现出低效的样本效率。这显著阻碍了此类代理在交互成本高的环境中的实用性（例如，当它们与人类交互或重置物理系统时）。尽管现有的一些LM代理架构集成了各种经验存储和反思机制，但它们对LM直接生成或推理完整反事实轨迹的能力利用有限。我们引入ECHO（通过回顾优化的经验整合），一种提示框架，它将强化学习中的回顾经验回放适应到语言模型代理中。ECHO为可能在失败尝试中实现的替代目标生成优化轨迹，从而有效地从不成功的交互中创建合成正例。我们的方法包含两个组成部分：一个回顾规则，利用语言模型本身来识别相关子目标并生成优化轨迹；一个更新规则，用于在内存中维护压缩的轨迹表示。我们在XMiniGrid的状态ful版本和PeopleJoinQA，一个协作信息收集企业模拟中评估了ECHO。在两个领域中，ECHO在 vanilla 语言代理基线上表现优异，最高可达80%；在XMiniGrid中，它还优于包括Reflexion和AWM在内的多种复杂代理架构，展示了通过更有效利用过去经验来更快适应新环境的能力。
英文摘要：
Language model (LM) agents deployed in novel environments often exhibit poor sample efficiency when learning from sequential interactions. This significantly hinders the usefulness of such agents in environments where interaction is costly (for example, when they interact with humans or reset physical systems). While a number of existing LM agent architectures incorporate various mechanisms for experience storage and reflection, they make limited use of LMs' abilities to directly generate or reason about full counterfactual trajectories. We introduce ECHO (Experience Consolidation via Hindsight Optimization), a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents. ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts, effectively creating synthetic positive examples from unsuccessful interactions. Our approach consists of two components: a hindsight rule that uses the language model itself to identify relevant subgoals and generate optimized trajectories, and an update rule that maintains compressed trajectory representations in memory. We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation. Across both domains, ECHO outperforms vanilla language agent baselines by up to 80%; in XMiniGrid, it also outperforms a number of sophisticated agent architectures including Reflexion and AWM, demonstrating faster adaptation to novel environments through more effective utilization of past experiences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
