arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07069cs.LGcs.AIcs.CL
收藏

使用大型语言模型自动化并加速带有奖励机器的强化学习

Using Large Language Models to Automate and Expedite Reinforcement Learning with Reward Machine

Arizona State University(亚利桑那州立大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Technical University of Dortmund(多特蒙德工业大学)

Shayan Meshkat Alsadat, Jean-Raphael Gaglione, Daniel Neider, Ufuk Topcu, Zhe Xu

AI总结:

提出LARL-RM算法,利用大型语言模型通过提示工程自动生成自动机高层知识,实现无需专家监督的完全闭环强化学习,理论保证收敛至最优策略,并在案例中提速30%。

摘要:

我们提出了LARL-RM(大型语言模型生成的自动机用于带有奖励机器的强化学习)算法,旨在使用自动机将高层知识编码到强化学习中以加速强化学习。我们的方法使用大型语言模型(LLM)通过提示工程获取高层领域特定知识,而不是直接向强化学习算法提供高层知识,因为后者需要专家来编码自动机。我们使用思维链和少样本方法进行提示工程,并证明我们的方法使用这些途径是有效的。此外,LARL-RM允许进行完全闭环的强化学习,无需专家指导和监督学习过程,因为LARL-RM可以直接使用LLM为当前任务生成所需的高层知识。我们还展示了算法收敛到最优策略的理论保证。我们通过在两个案例研究中实施该方法,证明LARL-RM将收敛速度提高了30%。

英文摘要:

We present LARL-RM (Large language model-generated Automaton for Reinforcement Learning with Reward Machine) algorithm in order to encode high-level knowledge into reinforcement learning using automaton to expedite the reinforcement learning. Our method uses Large Language Models (LLM) to obtain high-level domain-specific knowledge using prompt engineering instead of providing the reinforcement learning algorithm directly with the high-level knowledge which requires an expert to encode the automaton. We use chain-of-thought and few-shot methods for prompt engineering and demonstrate that our method works using these approaches. Additionally, LARL-RM allows for fully closed-loop reinforcement learning without the need for an expert to guide and supervise the learning since LARL-RM can use the LLM directly to generate the required high-level knowledge for the task at hand. We also show the theoretical guarantee of our algorithm to converge to an optimal policy. We demonstrate that LARL-RM speeds up the convergence by 30% by implementing our method in two case studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑