基于目标中心先验的3D特征蒸馏
3D Feature Distillation with Object-Centric Priors
AI总结：
针对2D CLIP特征提升至3D时存在的泛化性不足、依赖多视角等问题，提出基于目标中心先验的多视角特征融合策略，生成大规模合成数据集，实现单视角RGB-D下3D特征重建，提升关联精度与泛化性。
摘要：
将自然语言与物理世界关联是计算机视觉和机器人学中普遍存在的主题，具有广泛应用。近年来，CLIP等2D视觉语言模型因在2D图像开放词汇关联任务中的出色能力而广泛普及。近期研究试图通过特征蒸馏将2D CLIP特征提升至3D，但要么学习场景特定的神经场导致泛化性不足，要么聚焦于需多视角的室内场景扫描数据，这在机器人操作场景中不实用。此外，相关方法通常在像素级融合特征，假设所有视角同等重要。本研究表明这种方法会导致3D特征在关联精度和分割清晰度上表现欠佳。为此，我们提出一种多视角特征融合策略，利用目标中心先验基于语义信息消除无信息视角，并通过实例分割掩码在目标级融合特征。为蒸馏目标中心3D特征，我们生成大规模合成多视角杂乱桌面场景数据集，包含来自3300多个独特目标实例的15k场景并公开。实验表明，我们的方法能从单视角RGB-D重建具有更高关联能力和空间一致性的3D CLIP特征，无需测试时多视角假设。最后，我们证明该方法可泛化至新桌面领域，无需微调即可用于3D实例分割，并展示其在语言引导的杂乱环境机器人抓取中的实用性。
英文摘要：
Grounding natural language to the physical world is a ubiquitous topic with a wide range of applications in computer vision and robotics. Recently, 2D vision-language models such as CLIP have been widely popularized, due to their impressive capabilities for open-vocabulary grounding in 2D images. Recent works aim to elevate 2D CLIP features to 3D via feature distillation, but either learn neural fields that are scene-specific and hence lack generalization, or focus on indoor room scan data that require access to multiple camera views, which is not practical in robot manipulation scenarios. Additionally, related methods typically fuse features at pixel-level and assume that all camera views are equally informative. In this work, we show that this approach leads to sub-optimal 3D features, both in terms of grounding accuracy, as well as segmentation crispness. To alleviate this, we propose a multi-view feature fusion strategy that employs object-centric priors to eliminate uninformative views based on semantic information, and fuse features at object-level via instance segmentation masks. To distill our object-centric 3D features, we generate a large-scale synthetic multi-view dataset of cluttered tabletop scenes, spawning 15k scenes from over 3300 unique object instances, which we make publicly available. We show that our method reconstructs 3D CLIP features with improved grounding capacity and spatial consistency, while doing so from single-view RGB-D, thus departing from the assumption of multiple camera views at test time. Finally, we show that our approach can generalize to novel tabletop domains and be re-purposed for 3D instance segmentation without fine-tuning, and demonstrate its utility for language-guided robotic grasping in clutter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
