Retroformer：基于策略梯度优化的回顾式大型语言智能体
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
AI总结：
本文提出Retroformer框架，通过策略梯度从环境奖励中学习回顾式模型，自动优化语言智能体的提示，从而显著提升多步任务中的智能体性能。
摘要：
近几个月来，出现了一股强大的新趋势：大型语言模型（LLMs）被增强为自主语言智能体，能够独立执行面向目标的多步骤任务，而不仅仅是对人类用户的查询做出回应。然而，大多数现有语言智能体并未使用环境特定的奖励进行优化。尽管一些智能体通过语言反馈实现迭代改进，但它们的推理和规划方式并不兼容基于梯度的奖励学习。本文提出了一个原则性框架，通过学习回顾式模型来强化大型语言智能体，该模型通过策略梯度根据环境反馈自动调整语言智能体的提示。具体来说，我们提出的智能体架构从多个环境和任务中学习奖励，用于微调一个预训练语言模型，该模型通过总结先前失败尝试的根本原因并提出行动计划来优化语言智能体的提示。在多种任务上的实验结果表明，语言智能体随时间不断改进，并且我们的方法显著优于那些未能恰当利用环境梯度的基线方法。这表明使用策略梯度优化来改进语言智能体——我们相信我们的工作是该方向最早的研究之一——看起来很有前景，并且可以应用于优化智能体架构中的其他模型，以随时间提升智能体性能。
英文摘要：
Recent months have seen the emergence of a powerful new trend in which large language models (LLMs) are augmented to become autonomous language agents capable of performing objective oriented multi-step tasks on their own, rather than merely responding to queries from human users. Most existing language agents, however, are not optimized using environment-specific rewards. Although some agents enable iterative refinement through verbal feedback, they do not reason and plan in ways that are compatible with gradient-based learning from rewards. This paper introduces a principled framework for reinforcing large language agents by learning a retrospective model, which automatically tunes the language agent prompts from environment feedback through policy gradient. Specifically, our proposed agent architecture learns from rewards across multiple environments and tasks, for fine-tuning a pre-trained language model which refines the language agent prompt by summarizing the root cause of prior failed attempts and proposing action plans. Experimental results on various tasks demonstrate that the language agents improve over time and that our approach considerably outperforms baselines that do not properly leverage gradients from the environment. This demonstrates that using policy gradient optimization to improve language agents, for which we believe our work is one of the first, seems promising and can be applied to optimize other models in the agent architecture to enhance agent performances over time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
