arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07381cs.AIcs.LG
收藏

借助大型语言模型提示的世界模型用于目标达成

World Models with Hints of Large Language Models for Goal Achieving

The Ohio State University(俄亥俄州立大学);University of Maryland, College Park(马里兰大学学院公园分校);Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院);Shanghai AI Lab(上海人工智能实验室)

Zeyuan Liu, Ziyu Huan, Xiyao Wang, Jiafei Lyu, Jian Tao, Xiu Li, Furong Huang, Huazhe Xu

AI总结:

本文提出 DLLM,将 LLM 生成的提示性子目标融入世界模型推演并据此分配内在奖励,从而在多种稀疏奖励环境中显著提升长时序目标达成性能。

摘要:

强化学习在面对长时序任务和稀疏目标时举步维艰,原因在于人工设定奖励十分困难。现有方法通过添加内在奖励来解决这一问题,但在具有庞大状态和动作空间的长时序决策任务中,它们可能无法提供有意义的指导,缺乏有目的的探索。受人类认知启发,我们提出了一种名为 Dreaming with Large Language Models(DLLM)的新型多模态基于模型的强化学习方法。DLLM 将所提出的来自 LLM 的提示性子目标整合到模型推演中,以鼓励在具有挑战性的任务中发现并达成目标。通过在模型推演期间为符合语言模型所概述提示的样本赋予更高的内在奖励,DLLM 引导智能体进行有意义且高效的探索。大量实验表明,DLLM 在 HomeGrid、Crafter 和 Minecraft 等各种具有挑战性的稀疏奖励环境中,分别比近期方法高出 27.7%、21.1% 和 9.9%。

英文摘要:

Reinforcement learning struggles in the face of long-horizon tasks and sparse goals due to the difficulty in manual reward specification. While existing methods address this by adding intrinsic rewards, they may fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces, lacking purposeful exploration. Inspired by human cognition, we propose a new multi-modal model-based RL approach named Dreaming with Large Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the LLMs into the model rollouts to encourage goal discovery and reaching in challenging tasks. By assigning higher intrinsic rewards to samples that align with the hints outlined by the language model during model rollouts, DLLM guides the agent toward meaningful and efficient exploration. Extensive experiments demonstrate that the DLLM outperforms recent methods in various challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 27.7\%, 21.1\%, and 9.9\%, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑