3D-MVP：面向机器人操作的3D多视图预训练
3D-MVP: 3D Multiview Pretraining for Robotic Manipulation
AI总结：
针对现有机器人视觉预训练仅基于2D图像、缺乏3D场景理解的问题，提出3D-MVP方法，拆分RVT的视觉编码器并在3D数据集上用掩码自编码预训练，提升了虚拟操作任务性能与策略泛化性。
摘要：
近期研究表明，使用掩码自编码器（MAE）在第一人称视角数据集上进行视觉预训练，能够提升下游机器人任务的泛化能力。但这类方法仅在2D图像上开展预训练，而众多机器人应用都需要3D场景理解能力。本文提出3D-MVP，这是一种采用掩码自编码器实现3D多视图预训练的全新方法。我们利用Robotic View Transformer（机器人视图Transformer，简称RVT），该模型通过多视图Transformer理解3D场景并预测夹爪位姿动作。我们将RVT的多视图Transformer拆分为视觉编码器与动作解码器，并在Objaverse等大规模3D数据集上采用掩码自编码任务预训练其视觉编码器。我们在一组虚拟机器人操作任务上评估3D-MVP的性能，结果显示其表现优于基线方法。研究结果表明，具备3D感知能力的预训练是提升基于视觉的机器人操作策略泛化性的有效方向。项目主页：https://jasonqsy.github.io/3DMVP
英文摘要：
Recent works have shown that visual pretraining on egocentric datasets using masked autoencoders (MAE) can improve generalization for downstream robotics tasks. However, these approaches pretrain only on 2D images, while many robotics applications require 3D scene understanding. In this work, we propose 3D-MVP, a novel approach for 3D Multi-View Pretraining using masked autoencoders. We leverage Robotic View Transformer (RVT), which uses a multi-view transformer to understand the 3D scene and predict gripper pose actions. We split RVT's multi-view transformer into visual encoder and action decoder, and pretrain its visual encoder using masked autoencoding on large-scale 3D datasets such as Objaverse. We evaluate 3D-MVP on a suite of virtual robot manipulation tasks and demonstrate improved performance over baselines. Our results suggest that 3D-aware pretraining is a promising approach to improve generalization of vision-based robotic manipulation policies. Project site: https://jasonqsy.github.io/3DMVP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
