arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.07932cs.ROcs.AIcs.CV
收藏

什么对你重要?面向机器人学习的视觉表征对齐

What Matters to You? Towards Visual Representation Alignment for Robot Learning

UC Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学)

Ran Tian, Chenfeng Xu, Masayoshi Tomizuka, Jitendra Malik, Andrea Bajcsy

AI总结:

本文提出RAPL方法,通过人类反馈和最优传输对齐视觉表征与用户偏好,在X-MAGICAL和机器人操作中实现高样本效率和强零样本泛化的偏好对齐行为。

摘要:

当机器人为人服务时,需要优化与终端用户偏好一致的奖励。由于机器人将依赖RGB图像等原始感知输入,其奖励不可避免地会使用视觉表征。最近,使用预训练视觉模型的表征引起了广泛关注,但使其在机器人中发挥作用的关键是微调,而微调通常通过动力学预测或强制时间循环一致性等代理任务来完成。然而,所有这些代理任务都绕过了人类对其关注内容的输入,加剧了虚假相关性,最终导致机器人行为与用户偏好不一致。在这项工作中,我们提出机器人应利用人类反馈将其视觉表征与终端用户对齐,并解耦对任务重要的因素。我们提出了表征对齐的基于偏好的学习(RAPL),这是一种通过基于偏好的学习和最优传输来解决视觉表征对齐问题和视觉奖励学习问题的方法。在X-MAGICAL和机器人操作中的实验表明,RAPL的奖励能够以高样本效率持续生成符合偏好的机器人行为,并且当视觉表征从不同于机器人的具身形态中学习时,展现出强大的零样本泛化能力。

英文摘要:

When operating in service of people, robots need to optimize rewards aligned with end-user preferences. Since robots will rely on raw perceptual inputs like RGB images, their rewards will inevitably use visual representations. Recently there has been excitement in using representations from pre-trained visual models, but key to making these work in robotics is fine-tuning, which is typically done via proxy tasks like dynamics prediction or enforcing temporal cycle-consistency. However, all these proxy tasks bypass the human's input on what matters to them, exacerbating spurious correlations and ultimately leading to robot behaviors that are misaligned with user preferences. In this work, we propose that robots should leverage human feedback to align their visual representations with the end-user and disentangle what matters for the task. We propose Representation-Aligned Preference-based Learning (RAPL), a method for solving the visual representation alignment problem and visual reward learning problem through the lens of preference-based learning and optimal transport. Across experiments in X-MAGICAL and in robotic manipulation, we find that RAPL's reward consistently generates preferred robot behaviors with high sample efficiency, and shows strong zero-shot generalization when the visual representation is learned from a different embodiment than the robot's.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑