arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.16396cs.LG
收藏

GoalLadder: 基于视觉语言模型的增量目标发现

GoalLadder: Incremental Goal Discovery with Vision-Language Models

University of Oxford(牛津大学)

Alexey Zakharov, Shimon Whiteson

AI总结:

GoalLadder利用视觉语言模型在视觉环境中通过增量目标发现提升RL智能体性能,实现高成功率。

摘要:

自然语言可以提供一种简洁且易于人类理解的方式来指定强化学习(RL)任务。能够从语言指令中提取奖励的能力可以促进开发能够从人类指导中学习的机器人系统;然而,这仍然是一个具有挑战性的问题,尤其是在视觉环境中。现有方法要么依赖于非视觉环境表示,要么需要极大量的反馈,或者生成噪声且形状不良的奖励函数。在本文中,我们提出了一种新的方法,即GoalLadder,它利用视觉语言模型(VLMs)在视觉环境中从单一语言指令中训练RL智能体。GoalLadder通过逐步发现使智能体更接近完成自然语言指定任务的状态来工作。为此,它查询VLM以识别代表智能体任务进展改进的状态,并使用成对比较对它们进行排序。与先前工作不同,GoalLadder不完全信任VLM的反馈;相反,它使用ELO-based评分系统对潜在目标状态进行排序,从而减少噪声VLM反馈的负面影响。在训练过程中,智能体的任务是最小化在学习的嵌入空间中到最高排名目标的距离,该嵌入空间是在未标记的视觉数据上训练的。这一关键特性使我们能够绕过通常需要大量且准确的反馈来训练良好形状的奖励函数。我们证明了GoalLadder在经典控制和机器人操作环境中优于现有相关方法,其平均最终成功率约为95%,而最佳竞争对手仅为约45%。

英文摘要:

Natural language can offer a concise and human-interpretable means of specifying reinforcement learning (RL) tasks. The ability to extract rewards from a language instruction can enable the development of robotic systems that can learn from human guidance; however, it remains a challenging problem, especially in visual environments. Existing approaches that employ large, pretrained language models either rely on non-visual environment representations, require prohibitively large amounts of feedback, or generate noisy, ill-shaped reward functions. In this paper, we propose a novel method, GoalLadder, that leverages vision-language models (VLMs) to train RL agents from a single language instruction in visual environments. GoalLadder works by incrementally discovering states that bring the agent closer to completing a task specified in natural language. To do so, it queries a VLM to identify states that represent an improvement in agent's task progress and to rank them using pairwise comparisons. Unlike prior work, GoalLadder does not trust VLM's feedback completely; instead, it uses it to rank potential goal states using an ELO-based rating system, thus reducing the detrimental effects of noisy VLM feedback. Over the course of training, the agent is tasked with minimising the distance to the top-ranked goal in a learned embedding space, which is trained on unlabelled visual data. This key feature allows us to bypass the need for abundant and accurate feedback typically required to train a well-shaped reward function. We demonstrate that GoalLadder outperforms existing related methods on classic control and robotic manipulation environments with the average final success rate of $\sim$95% compared to only $\sim$45% of the best competitor.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑