arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.11366cs.AIcs.CLcs.LG
收藏

Reflexion:具有语言强化学习的语言智能体

Reflexion: Language Agents with Verbal Reinforcement Learning

Northeastern University(东北大学);Massachusetts Institute of Technology(麻省理工学院);Princeton University(普林斯顿大学)

Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao

AI总结:

Reflexion框架通过语言反馈而非权重更新强化语言智能体,利用口头反思和情节记忆提升决策,在HumanEval上达到91%准确率,超越GPT-4的80%。

摘要:

大型语言模型(LLMs)越来越多地被用作与外部环境(如游戏、编译器、API)交互的目标驱动智能体。然而,这些语言智能体很难从试错中快速有效地学习,因为传统的强化学习方法需要大量的训练样本和昂贵的模型微调。我们提出了Reflexion,一种新颖的框架,通过语言反馈而非更新权重来强化语言智能体。具体而言,Reflexion智能体口头反思任务反馈信号,然后在情节记忆缓冲区中维护自己的反思文本,以在后续试验中诱导更好的决策。Reflexion足够灵活,可以整合各种类型(标量值或自由形式语言)和来源(外部或内部模拟)的反馈信号,并在各种任务(顺序决策、编码、语言推理)上相对于基线智能体获得了显著改进。例如,Reflexion在HumanEval编码基准上实现了91%的pass@1准确率,超过了之前最先进的GPT-4(达到80%)。我们还使用不同的反馈信号、反馈整合方法和智能体类型进行了消融和分析研究,并提供了关于它们如何影响性能的见解。

英文摘要:

Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑