Cues3D：在开放词汇3D全景分割中释放单一NeRF实现一致且唯一实例的力量
Cues3D: Unleashing the Power of Sole NeRF for Consistent and Unique Instances in Open-Vocabulary 3D Panoptic Segmentation
AI总结：
针对开放词汇3D全景分割中NeRF方法难以维持跨视图实例一致性的问题，提出Cues3D，仅依赖NeRF隐式几何实现全局唯一实例ID，通过三阶段训练和实例消歧在多数据集上取得优异性能。
摘要：
开放词汇3D全景分割近期已成为一个重要趋势。目前表现顶尖的方法通常将2D分割与几何感知的3D基元相结合。然而，若缺乏高保真3D点云（例如基于Neural Radiance Field（NeRF）的方法），这种优势便会丧失。这些方法受限于在部分观测间维持一致性的能力不足。为解决此问题，近期研究利用对比损失或跨视图关联预处理来实现视图共识。与它们不同，我们提出了Cues3D，这是一种仅依赖NeRF而非预关联的紧凑方法。其核心思想是，NeRF的隐式3D场本质上建立了一个全局一致的几何结构，从而无需显式的跨视图监督即可有效区分物体。我们为NeRF提出了一种三阶段训练框架，即初始化-消歧-细化，通过利用初始学习的知识来纠正实例ID。此外，还提出了一种实例消歧方法，用于匹配NeRF渲染的3D掩码并确保全局唯一的3D实例身份。借助Cues3D，我们利用平衡版本的NeRF，为跨视图的每个物体获取了高度一致且唯一的3D实例ID。我们在ScanNet v2、ScanNet200、ScanNet++和Replica数据集上进行了3D实例、全景及语义分割任务的实验。Cues3D优于其他基于2D图像的方法，并与最新的基于2D-3D融合的方法相媲美，甚至在使用额外3D点云时超越了它们。代码链接见附录，并将发布在github。
英文摘要：
Open-vocabulary 3D panoptic segmentation has recently emerged as a significant trend. Top-performing methods currently integrate 2D segmentation with geometry-aware 3D primitives. However, the advantage would be lost without high-fidelity 3D point clouds, such as methods based on Neural Radiance Field (NeRF). These methods are limited by the insufficient capacity to maintain consistency across partial observations. To address this, recent works have utilized contrastive loss or cross-view association pre-processing for view consensus. In contrast to them, we present Cues3D, a compact approach that relies solely on NeRF instead of pre-associations. The core idea is that NeRF's implicit 3D field inherently establishes a globally consistent geometry, enabling effective object distinction without explicit cross-view supervision. We propose a three-phase training framework for NeRF, initialization-disambiguation-refinement, whereby the instance IDs are corrected using the initially-learned knowledge. Additionally, an instance disambiguation method is proposed to match NeRF-rendered 3D masks and ensure globally unique 3D instance identities. With the aid of Cues3D, we obtain highly consistent and unique 3D instance ID for each object across views with a balanced version of NeRF. Our experiments are conducted on ScanNet v2, ScanNet200, ScanNet++, and Replica datasets for 3D instance, panoptic, and semantic segmentation tasks. Cues3D outperforms other 2D image-based methods and competes with the latest 2D-3D merging based methods, while even surpassing them when using additional 3D point clouds. The code link could be found in the appendix and will be released on \href{https://github.com/mRobotit/Cues3D}{github}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
