arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07157cs.CLcs.AIcs.LG
收藏

自然语言强化学习

Natural Language Reinforcement Learning

University College London(伦敦大学学院);University of Surrey(萨里大学);Shanghai Jiao Tong University(上海交通大学)

Xidong Feng, Ziyu Wan, Mengyue Yang, Ziyan Wang, Girish A. Koushik, Yali Du, Ying Wen, Jun Wang

AI总结:

针对强化学习样本效率低和可解释性差等问题,提出自然语言强化学习(NLRL),结合大语言模型在自然语言空间重新定义RL核心概念,实验证明其有效且具可解释性。

摘要:

强化学习(RL)在决策任务的学习策略方面展现出卓越能力。然而,RL常受限于样本效率低、缺乏可解释性和监督信号稀疏等问题。为解决这些局限,我们从人类学习过程中汲取灵感,提出了自然语言强化学习(NLRL),创新性地将RL原理与自然语言表示相结合。具体而言,NLRL在自然语言空间中重新定义了任务目标、策略、价值函数、Bellman方程和策略迭代等RL概念。我们展示了如何利用GPT-4等大型语言模型(LLMs)的最新进展来实际实现NLRL。在表格MDP上的初步实验证明了NLRL框架的有效性、效率以及可解释性。

英文摘要:

Reinforcement Learning (RL) has shown remarkable abilities in learning policies for decision-making tasks. However, RL is often hindered by issues such as low sample efficiency, lack of interpretability, and sparse supervision signals. To tackle these limitations, we take inspiration from the human learning process and introduce Natural Language Reinforcement Learning (NLRL), which innovatively combines RL principles with natural language representation. Specifically, NLRL redefines RL concepts like task objectives, policy, value function, Bellman equation, and policy iteration in natural language space. We present how NLRL can be practically implemented with the latest advancements in large language models (LLMs) like GPT-4. Initial experiments over tabular MDPs demonstrate the effectiveness, efficiency, and also interpretability of the NLRL framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑