arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01874cs.CLcs.AIcs.LGcs.RO
收藏

RL/LLM 分类树:综述强化学习与大型语言模型之间的协同作用

The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models

Microsoft(微软公司)

Moschoula Pternea, Prerna Singh, Abir Chakraborty, Yagna Oruganti, Mirco Milletari, Sayli Bapat, Kebei Jiang

AI总结:

本文综述了强化学习与大型语言模型的结合研究,提出包含RL4LLM、LLM4RL和RL+LLM三个主要类别的分类法,据此探讨两者协同的动机、成功原因、潜在不足及替代方法。

摘要:

在本研究中,我们回顾了结合强化学习(RL)和大型语言模型(LLMs)的研究,这两个领域的发展动力均源于深度神经网络的发展。我们基于两种模型类型的交互方式,提出了一种包含三个主要类别的新颖分类法。第一类 RL4LLM 包括利用 RL 提升 LLM 在与自然语言处理相关任务上的性能的研究。RL4LLM 根据是使用 RL 直接微调现有 LLM 还是改进 LLM 的提示而分为两个子类别。在第二类 LLM4RL 中,LLM 协助训练执行本质上与自然语言无关任务的 RL 模型。我们根据 LLM 协助或替代的 RL 训练框架的组件(即奖励塑造、目标生成和策略函数)进一步细分 LLM4RL。最后,在第三类 RL+LLM 中,LLM 和 RL 智能体被嵌入到一个共同的规划框架中,且两者均不对彼此的训练或微调做出贡献。我们进一步分支此类,以区分有无自然语言反馈的研究。我们使用该分类法探索 LLM 与 RL 协同背后的动机,并解释其成功的原因,同时指出潜在不足和需要进一步研究的领域,以及服务于相同目标的替代方法。

英文摘要:

In this work, we review research studies that combine Reinforcement Learning (RL) and Large Language Models (LLMs), two areas that owe their momentum to the development of deep neural networks. We propose a novel taxonomy of three main classes based on the way that the two model types interact with each other. The first class, RL4LLM, includes studies where RL is leveraged to improve the performance of LLMs on tasks related to Natural Language Processing. L4LLM is divided into two sub-categories depending on whether RL is used to directly fine-tune an existing LLM or to improve the prompt of the LLM. In the second class, LLM4RL, an LLM assists the training of an RL model that performs a task that is not inherently related to natural language. We further break down LLM4RL based on the component of the RL training framework that the LLM assists or replaces, namely reward shaping, goal generation, and policy function. Finally, in the third class, RL+LLM, an LLM and an RL agent are embedded in a common planning framework without either of them contributing to training or fine-tuning of the other. We further branch this class to distinguish between studies with and without natural language feedback. We use this taxonomy to explore the motivations behind the synergy of LLMs and RL and explain the reasons for its success, while pinpointing potential shortcomings and areas where further research is needed, as well as alternative methodologies that serve the same goal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑