CLAMP: 基于对比学习的3D多视角动作条件机器人操控预训练
CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining
AI总结：
CLAMP通过3D点云和机器人动作进行预训练，利用对比学习提升机器人操控精度与效率，优于现有基线方法。
摘要：
利用预训练的2D图像表示在行为克隆策略中取得了巨大成功，已成为机器人操控的标准方法。然而，此类表示未能捕捉到物体和场景的3D空间信息，这对于精确操控至关重要。本文介绍了一种新的3D预训练框架CLAMP，利用点云和机器人动作。从RGB-D图像和相机外参计算出的合并点云中，重新渲染多视角四通道图像观察，包括动态腕部视角，以提供更清晰的目标物体视图，用于高精度操控任务。预训练的编码器通过对比学习学习将物体的3D几何和位置信息与机器人动作模式关联起来，基于大规模模拟机器人轨迹。在编码器预训练过程中，我们预训练了一个扩散策略来初始化策略权重，这对于提高微调样本效率和性能至关重要。预训练后，我们使用学习到的图像和动作表示在有限的任务演示上微调策略。我们证明了这种预训练和微调设计显著提高了学习效率和在未见任务上的策略性能。此外，我们展示了CLAMP在六个模拟任务和五个现实任务中均优于现有最先进的基线方法。项目网站和视频可在https://clamp3d.github.io/CLAMP/上找到。
英文摘要：
Leveraging pre-trained 2D image representations in behavior cloning policies has achieved great success and has become a standard approach for robotic manipulation. However, such representations fail to capture the 3D spatial information about objects and scenes that is essential for precise manipulation. In this work, we introduce Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining (CLAMP), a novel 3D pre-training framework that utilizes point clouds and robot actions. From the merged point cloud computed from RGB-D images and camera extrinsics, we re-render multi-view four-channel image observations with depth and 3D coordinates, including dynamic wrist views, to provide clearer views of target objects for high-precision manipulation tasks. The pre-trained encoders learn to associate the 3D geometric and positional information of objects with robot action patterns via contrastive learning on large-scale simulated robot trajectories. During encoder pre-training, we pre-train a Diffusion Policy to initialize the policy weights for fine-tuning, which is essential for improving fine-tuning sample efficiency and performance. After pre-training, we fine-tune the policy on a limited amount of task demonstrations using the learned image and action representations. We demonstrate that this pre-training and fine-tuning design substantially improves learning efficiency and policy performance on unseen tasks. Furthermore, we show that CLAMP outperforms state-of-the-art baselines across six simulated tasks and five real-world tasks. The project website and videos can be found at https://clamp3d.github.io/CLAMP/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
