OpenIns3D：面向3D开放词汇实例分割的捕捉与检索方法
OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation
AI总结：
本研究提出仅3D输入的OpenIns3D框架，通过掩码-捕捉-检索机制实现3D开放词汇场景理解，在多类任务与数据集上达SOTA，支持灵活切换2D检测器，结合LLM模型可处理复杂推理文本查询。
摘要：
在本研究中，我们提出了OpenIns3D，这是一种全新的仅以3D数据为输入的框架，用于3D开放词汇场景理解。OpenIns3D框架采用“Mask-Snap-Lookup（掩码-捕捉-检索）”机制：“Mask（掩码）”模块负责在3D点云中学习类别无关的掩码提议；“Snap（捕捉）”模块生成多尺度的合成场景级图像，并利用2D视觉-语言模型提取感兴趣的目标；“Lookup（检索）”模块则对“Snap”的输出结果进行搜索，为生成的掩码提议分配类别名称。该方法尽管结构简单，却在室内和室外数据集上的各类3D开放词汇任务（包括识别、目标检测与实例分割）中均取得了当前最优性能。此外，OpenIns3D无需重新训练即可轻松切换不同的2D检测器，当与性能强大的2D开放世界模型集成时，能在场景理解任务中取得优异结果。不仅如此，当结合基于大语言模型（LLM）的2D模型时，OpenIns3D展现出了出色的能力，可理解并处理需要复杂推理与现实世界知识的高度复杂文本查询。项目页面：https://zheninghuang.github.io/OpenIns3D/
英文摘要：
In this work, we introduce OpenIns3D, a new 3D-input-only framework for 3D open-vocabulary scene understanding. The OpenIns3D framework employs a "Mask-Snap-Lookup" scheme. The "Mask" module learns class-agnostic mask proposals in 3D point clouds, the "Snap" module generates synthetic scene-level images at multiple scales and leverages 2D vision-language models to extract interesting objects, and the "Lookup" module searches through the outcomes of "Snap" to assign category names to the proposed masks. This approach, yet simple, achieves state-of-the-art performance across a wide range of 3D open-vocabulary tasks, including recognition, object detection, and instance segmentation, on both indoor and outdoor datasets. Moreover, OpenIns3D facilitates effortless switching between different 2D detectors without requiring retraining. When integrated with powerful 2D open-world models, it achieves excellent results in scene understanding tasks. Furthermore, when combined with LLM-powered 2D models, OpenIns3D exhibits an impressive capability to comprehend and process highly complex text queries that demand intricate reasoning and real-world knowledge. Project page: https://zheninghuang.github.io/OpenIns3D/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
