发表机构
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
SenseFuse提出一种无标签融合方法，结合2D图像与3D形状编码器，通过自适应权重优化开放词汇3D实例分割的掩码标注，显著提升准确性与鲁棒性。
AI 中文摘要
开放词汇场景理解是机器人技术的基础，为空间推理和物体操作奠定基础。虽然封闭词汇的3D实例分割严重依赖3D形状信息，但最先进的开放词汇方法在掩码标注阶段仍主要局限于2D图像特征或图像蒸馏表示。本文提出SenseFuse，一种无标签融合方法，平衡2D图像和3D形状编码器以实现鲁棒的开放词汇3D实例分割，仅优化现有流程中的掩码标注阶段。我们发现2D图像和3D形状编码器表现出大部分不重叠的失败模式，很少共享相同的错误标签，而两个2D图像编码器则经常重复相同的错误。这种独特的行为使得2D和3D组合天然互补。我们引入一种自适应机制，选择场景级融合权重以最大化无标签敏感性度量，该度量直接从单个场景的未标注提议中在毫秒级估计。SenseFuse在ScanNet200、Replica和ScanNet++的每个评估设置中均提高了标注准确性，恢复了使用oracle权重可获得增益的67-100%（中位数93%），并在22个报告设置中的21个中提高了实例AP。代码可在以下URL获取。
英文摘要
Open-vocabulary scene understanding is fundamental for robotics, laying the groundwork for spatial reasoning and object manipulation. While closed-vocabulary 3D instance segmentation heavily leverages 3D shape information, state-of-the-art open-vocabulary methods remain predominantly restricted to 2D image features or image-distilled representations during mask labeling. In this paper, we propose SenseFuse, a label-free fusion method that balances 2D image and 3D shape encoders for robust open-vocabulary 3D instance segmentation, refining only the mask-labeling stage of existing pipelines. We reveal that 2D image and 3D shape encoders exhibit largely disjoint failure patterns and rarely share identical wrong labels, whereas two 2D image encoders frequently repeat the same errors. This distinct behavior makes the 2D and 3D pair inherently complementary. We introduce an adaptive mechanism that selects a scene-level fusion weight to maximize a label-free sensitivity measure, estimated directly from a single scene's unlabeled proposals in milliseconds. SenseFuse improves labeling accuracy in every evaluated setting across ScanNet200, Replica, and ScanNet++, recovering 67-100% (median 93%) of the gain achievable with an oracle weight, and it raises instance AP in 21 of 22 reported settings. Code is available at https://github.com/hanes1207/SenseFuse.
Comments8 pages, 6 figures. Code: https://github.com/hanes1207/SenseFuse
