arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.07929cs.CLcs.AI
收藏

大语言模型是半参数强化学习智能体

Large Language Models Are Semi-Parametric Reinforcement Learning Agents

Shanghai Jiao Tong University(上海交通大学);Suzhou Laboratory(苏州实验室)

Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, Kai Yu

AI总结:

本文提出 REMEMBERER 框架,通过为 LLM 配备长期经验记忆并结合 RLEM 更新机制,使智能体无需微调参数即可从成功与失败经验中持续演化,在两个 RL 任务集上成功率分别超越 SOTA 4% 和 2%。

摘要:

受认知科学中关于人类记忆与推理机制的见解启发,本文提出了一种新颖的、可演化的基于大语言模型(LLM)的智能体框架,称为 REMEMBERER。通过为 LLM 配备长期经验记忆,REMEMBERER 能够利用来自过去回合的经验,即使面对不同的任务目标也能发挥作用,这优于使用固定示例或仅配备瞬时工作记忆的基于 LLM 的智能体。我们进一步引入带经验记忆的强化学习(RLEM)来更新记忆。因此,整个系统能够从成功和失败的经验中学习,并在不微调 LLM 参数的情况下演化其能力。通过这种方式,所提出的 REMEMBERER 构成了一个半参数强化学习智能体。在两个强化学习任务集上进行了大量实验以评估所提出的框架。在不同初始化和训练集下的平均结果显示,REMEMBERER 在两个任务集上的成功率分别比先前的最优方法(SOTA)高出 4% 和 2%,证明了 REMEMBERER 的优越性和鲁棒性。

英文摘要:

Inspired by the insights in cognitive science with respect to human memory and reasoning mechanism, a novel evolvable LLM-based (Large Language Model) agent framework is proposed as REMEMBERER. By equipping the LLM with a long-term experience memory, REMEMBERER is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars or equipped with a transient working memory. We further introduce Reinforcement Learning with Experience Memory (RLEM) to update the memory. Thus, the whole system can learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters of the LLM. In this way, the proposed REMEMBERER constitutes a semi-parametric RL agent. Extensive experiments are conducted on two RL task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by 4% and 2% for the success rate on two task sets and demonstrate the superiority and robustness of REMEMBERER.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑