arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.03402cs.AIcs.RO
收藏

游戏开始:迈向作为强化学习实验者的语言模型

Game On: Towards Language Models as RL Experimenters

Google DeepMind(谷歌DeepMind)

Jingwei Zhang, Thomas Lampe, Abbas Abdolmaleki, Jost Tobias Springenberg, Martin Riedmiller

AI总结:

本文提出一种利用视觉语言模型(VLM)在强化学习全实验周期中自动监控进展、生成任务课程并检索技能的智能体架构,原型实验表明其能有效促进机器人控制策略的迭代学习。

摘要:

我们提出了一种智能体架构,用于自动化常见强化学习实验工作流的部分环节,以实现具身智能体对控制领域的自动化掌握。为此,该架构利用视觉语言模型(VLM)来执行通常需要人类实验者具备的部分能力,包括监控和分析实验进展、基于智能体过去的成功与失败提出新任务、将任务分解为一系列子任务(技能),以及检索要执行的技能——从而使我们的系统能够为学习构建自动化课程。我们认为,这是首批提出在强化学习的完整实验周期中利用VLM的系统之一。我们提供了该系统的首个原型,并考察了当前模型和技术在实现所需自动化水平方面的可行性。为此,我们使用一个标准的Gemini模型,不进行额外微调,为语言条件化的Actor-Critic算法提供技能课程,以引导数据收集,从而帮助学习新技能。以这种方式收集的数据被证明对机器人领域中控制策略的学习和迭代改进是有用的。对该系统构建不断增长的技能库以及判断这些技能训练进展能力的进一步检验也显示出有希望的结果,表明所提出的架构为具身智能体完全自动化掌握任务和领域提供了一种潜在的方案。

英文摘要:

We propose an agent architecture that automates parts of the common reinforcement learning experiment workflow, to enable automated mastery of control domains for embodied agents. To do so, it leverages a VLM to perform some of the capabilities normally required of a human experimenter, including the monitoring and analysis of experiment progress, the proposition of new tasks based on past successes and failures of the agent, decomposing tasks into a sequence of subtasks (skills), and retrieval of the skill to execute - enabling our system to build automated curricula for learning. We believe this is one of the first proposals for a system that leverages a VLM throughout the full experiment cycle of reinforcement learning. We provide a first prototype of this system, and examine the feasibility of current models and techniques for the desired level of automation. For this, we use a standard Gemini model, without additional fine-tuning, to provide a curriculum of skills to a language-conditioned Actor-Critic algorithm, in order to steer data collection so as to aid learning new skills. Data collected in this way is shown to be useful for learning and iteratively improving control policies in a robotics domain. Additional examination of the ability of the system to build a growing library of skills, and to judge the progress of the training of those skills, also shows promising results, suggesting that the proposed architecture provides a potential recipe for fully automated mastery of tasks and domains for embodied agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑