ViViDex：从人类视频中学习基于视觉的灵巧操作
ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos
浏览论文内容
中文总结 AI 辅助
本文提出ViViDex框架，通过强化学习结合轨迹引导奖励训练基于状态的策略，再利用其成功片段训练无需特权信息的统一视觉策略，解决了人类视频中噪声和特权信息依赖的问题，在三个灵巧操作任务上超越现有方法。
中文摘要 AI 辅助
在这项工作中，我们旨在为多指机器人手学习一种统一的基于视觉的策略，以操纵处于不同姿态的多种物体。尽管先前的工作已经展示了使用人类视频进行策略学习的益处，但性能提升受到了估计轨迹中噪声的限制。此外，对特权物体信息（如真实物体状态）的依赖进一步限制了其在现实场景中的适用性。为了解决这些局限性，我们提出了一个新的框架ViViDex，以改进从人类视频中学习基于视觉的策略。它首先使用带有轨迹引导奖励的强化学习为每个视频训练基于状态的策略，从视频中获取既视觉自然又物理上合理的轨迹。然后，我们推出基于状态策略的成功片段，并在不使用任何特权信息的情况下训练统一的视觉策略。我们提出坐标变换以进一步增强视觉点云表示，并比较了行为克隆和扩散策略在视觉策略训练中的表现。在模拟和真实机器人上的实验表明，ViViDex在三个灵巧操作任务上优于最先进的方法。
英文摘要
In this work, we aim to learn a unified vision-based policy for multi-fingered robot hands to manipulate a variety of objects in diverse poses. Though prior work has shown benefits of using human videos for policy learning, performance gains have been limited by the noise in estimated trajectories. Moreover, reliance on privileged object information such as ground-truth object states further limits the applicability in realistic scenarios. To address these limitations, we propose a new framework ViViDex to improve vision-based policy learning from human videos. It first uses reinforcement learning with trajectory guided rewards to train state-based policies for each video, obtaining both visually natural and physically plausible trajectories from the video. We then rollout successful episodes from state-based policies and train a unified visual policy without using any privileged information. We propose coordinate transformation to further enhance the visual point cloud representation, and compare behavior cloning and diffusion policy for the visual policy training. Experiments both in simulation and on the real robot demonstrate that ViViDex outperforms state-of-the-art approaches on three dexterous manipulation tasks.
发表机构
- Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
机构由 AI 辅助整理，请以论文原文为准。
