Open-Fusion：实时开放词汇3D建图与可查询场景表示
Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation
AI总结：
针对现有3D建图方法依赖预定义概念、生成语义地图耗时久的问题，提出Open-Fusion方法，结合VLFM与TSDF，通过改进的特征匹配实现无额外3D训练的实时开放词汇3D建图，在ScanNet上性能优于主流零样本方法。
摘要：
精确的3D环境建图在机器人技术中至关重要。现有方法通常在训练阶段依赖预定义概念，或生成语义地图时耗时较长。本文提出Open-Fusion，这是一种利用RGB-D数据实现实时开放词汇3D建图与可查询场景表示的突破性方法。Open-Fusion借助预训练视觉语言基础模型（VLFM）实现开放集语义理解，并采用截断符号距离函数（TSDF）完成快速3D场景重建。我们通过VLFM提取基于区域的嵌入及其对应的置信度图，再借助改进的基于匈牙利算法的特征匹配机制，将这些信息与TSDF提供的3D知识相融合。值得注意的是，Open-Fusion无需额外3D训练，即可实现出色的开放词汇无标注3D分割。在ScanNet数据集上与主流零样本方法的基准测试对比，凸显了Open-Fusion的优越性。此外，它无缝结合了基于区域的VLFM与TSDF的优势，可实现包含物体概念与开放世界语义的实时3D场景理解。我们建议读者访问项目页面https://uark-aicv.github.io/OpenFusion查看演示。
英文摘要：
Precise 3D environmental mapping is pivotal in robotics. Existing methods often rely on predefined concepts during training or are time-intensive when generating semantic maps. This paper presents Open-Fusion, a groundbreaking approach for real-time open-vocabulary 3D mapping and queryable scene representation using RGB-D data. Open-Fusion harnesses the power of a pre-trained vision-language foundation model (VLFM) for open-set semantic comprehension and employs the Truncated Signed Distance Function (TSDF) for swift 3D scene reconstruction. By leveraging the VLFM, we extract region-based embeddings and their associated confidence maps. These are then integrated with 3D knowledge from TSDF using an enhanced Hungarian-based feature-matching mechanism. Notably, Open-Fusion delivers outstanding annotation-free 3D segmentation for open-vocabulary without necessitating additional 3D training. Benchmark tests on the ScanNet dataset against leading zero-shot methods highlight Open-Fusion's superiority. Furthermore, it seamlessly combines the strengths of region-based VLFM and TSDF, facilitating real-time 3D scene comprehension that includes object concepts and open-world semantics. We encourage the readers to view the demos on our project page: https://uark-aicv.github.io/OpenFusion
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
