SAC-GLAM：通过软演员-评论员和回顾重标记改进LLM代理的在线强化学习
SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling
AI总结：
SAC-GLAM通过结合软演员-评论员算法和回顾重标记，改进LLM代理的在线强化学习，提升其在复杂环境中的策略学习能力。
摘要：
近年来，大型语言模型（LLMs）不仅作为生成模型，还作为解决文本序列决策任务的代理而努力。当面对复杂环境，其零样本能力不足时，最近的研究表明，可以使用在线强化学习（RL）来帮助LLM代理发现和学习高效的策略。然而，大多数先前的工作都局限于策略算法，这大大减少了此类代理在探索和利用方面可以使用的策略范围，例如经验回放和回顾重标记。然而，这些方法可能对LLM学习代理至关重要，特别是在设计自主内在动机代理时，这些代理会采样并追求自己的目标（即自主动机代理）。本文提出了并研究了软演员-评论员和回顾重标记对LLM代理的适应。我们的方法不仅为能够在线学习的自主LLM代理铺平了道路，而且在更经典的多目标RL环境中也能优于策略算法。
英文摘要：
The past years have seen Large Language Models (LLMs) strive not only as generative models but also as agents solving textual sequential decision-making tasks. When facing complex environments where their zero-shot abilities are insufficient, recent work showed online Reinforcement Learning (RL) could be used for the LLM agent to discover and learn efficient strategies interactively. However, most prior work sticks to on-policy algorithms, which greatly reduces the scope of methods such agents could use for both exploration and exploitation, such as experience replay and hindsight relabeling. Yet, such methods may be key for LLM learning agents, and in particular when designing autonomous intrinsically motivated agents sampling and pursuing their own goals (i.e. autotelic agents). This paper presents and studies an adaptation of Soft Actor-Critic and hindsight relabeling to LLM agents. Our method not only paves the path towards autotelic LLM agents that learn online but can also outperform on-policy methods in more classic multi-goal RL environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
