通过视觉提示实现可供性引导的强化学习
Affordance-Guided Reinforcement Learning via Visual Prompting
AI总结：
本文提出 KAGI，利用 VLM 通过关键点生成的可供性推理构造稠密奖励，引导自主强化学习，在真实操作任务中提升样本效率并减少演示依赖。
摘要：
配备强化学习（RL）的机器人有可能仅依靠奖励信号学习广泛的技能。然而，为通用操作任务获取鲁棒且稠密的奖励信号仍然是一项挑战。现有的基于学习的方法需要大量数据，例如成功和失败的人类演示，才能学习特定任务的奖励函数。近来，越来越多的大型多模态基础模型被用于机器人领域，这些模型能够在物理环境中执行视觉推理，并为操作任务生成粗略的机器人运动。受这一系列能力的启发，本文提出 Keypoint-based Affordance Guidance for Improvements（KAGI），这是一种利用视觉-语言模型（VLM）塑造奖励以进行自主 RL 的方法。最先进的 VLM 已展示出通过关键点进行令人印象深刻的可供性零样本推理的能力，作者利用这些关键点定义稠密奖励，以引导自主机器人学习。在由自然语言描述指定的多种真实世界操作任务上，KAGI 提高了自主 RL 的样本效率，并能够在 30K 次在线微调步骤内成功完成任务。此外，作者还证明了 KAGI 对预训练所用领域内演示数量减少的鲁棒性：在 45K 次在线微调步骤中达到相近性能。项目网站：https://sites.google.com/view/affordance-guided-rl
英文摘要：
Robots equipped with reinforcement learning (RL) have the potential to learn a wide range of skills solely from a reward signal. However, obtaining a robust and dense reward signal for general manipulation tasks remains a challenge. Existing learning-based approaches require significant data, such as human demonstrations of success and failure, to learn task-specific reward functions. Recently, there is also a growing adoption of large multi-modal foundation models for robotics that can perform visual reasoning in physical contexts and generate coarse robot motions for manipulation tasks. Motivated by this range of capability, in this work, we present Keypoint-based Affordance Guidance for Improvements (KAGI), a method leveraging rewards shaped by vision-language models (VLMs) for autonomous RL. State-of-the-art VLMs have demonstrated impressive zero-shot reasoning about affordances through keypoints, and we use these to define dense rewards that guide autonomous robotic learning. On diverse real-world manipulation tasks specified by natural language descriptions, KAGI improves the sample efficiency of autonomous RL and enables successful task completion in 30K online fine-tuning steps. Additionally, we demonstrate the robustness of KAGI to reductions in the number of in-domain demonstrations used for pre-training, reaching similar performance in 45K online fine-tuning steps. Project website: https://sites.google.com/view/affordance-guided-rl
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
