Reasoning3D —— 3D中的定位与推理：基于大型视觉-语言模型的细粒度零样本开放词汇3D推理部件分割
Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models
AI总结：
本文提出零样本3D推理分割新任务及基线方法Reasoning3D，利用大型语言模型驱动的预训练2D分割网络，实现对3D网格部件的细粒度定位与自然语言解释，为多领域提供免训练的通用基线。
摘要：
在本文中，我们引入了一项新任务：用于物体部件搜索与定位的零样本3D推理分割，这是一种超越以往特定类别3D语义分割、3D实例分割和开放词汇3D分割限制的3D分割新范式。我们设计了一个简单的基线方法Reasoning3D，它能够理解并执行复杂命令，对3D网格的特定部件进行（细粒度）分割，具备上下文感知能力，并为交互式分割提供推理答案。具体而言，Reasoning3D利用现成的预训练2D分割网络，并由大型语言模型（LLMs）驱动，以零样本方式解释用户输入查询。先前的研究表明，广泛的预训练赋予基础模型先验世界知识，使其能够理解复杂命令，我们可以利用这一能力在3D数据集有限的情况下（源高效）实现“分割一切”。实验表明，我们的方法具有泛化能力，能够基于隐式文本查询有效定位并高亮3D物体（3D网格中）的部件，包括铰接3D物体和真实世界扫描数据。我们的方法还能生成与这些3D模型及其分解相对应的自然语言解释。此外，我们这种免训练方法允许快速部署，并可作为未来在机器人、物体操作、部件装配、自动驾驶应用、增强现实与虚拟现实（AR/VR）以及医疗应用等领域中部件级3D（语义）物体理解研究的可行通用基线。代码、模型权重、部署指南和评估协议见：http://tianrun-chen.github.io/Reason3D/
英文摘要：
In this paper, we introduce a new task: Zero-Shot 3D Reasoning Segmentation for parts searching and localization for objects, which is a new paradigm to 3D segmentation that transcends limitations for previous category-specific 3D semantic segmentation, 3D instance segmentation, and open-vocabulary 3D segmentation. We design a simple baseline method, Reasoning3D, with the capability to understand and execute complex commands for (fine-grained) segmenting specific parts for 3D meshes with contextual awareness and reasoned answers for interactive segmentation. Specifically, Reasoning3D leverages an off-the-shelf pre-trained 2D segmentation network, powered by Large Language Models (LLMs), to interpret user input queries in a zero-shot manner. Previous research have shown that extensive pre-training endows foundation models with prior world knowledge, enabling them to comprehend complex commands, a capability we can harness to "segment anything" in 3D with limited 3D datasets (source efficient). Experimentation reveals that our approach is generalizable and can effectively localize and highlight parts of 3D objects (in 3D mesh) based on implicit textual queries, including these articulated 3d objects and real-world scanned data. Our method can also generate natural language explanations corresponding to these 3D models and the decomposition. Moreover, our training-free approach allows rapid deployment and serves as a viable universal baseline for future research of part-level 3d (semantic) object understanding in various fields including robotics, object manipulation, part assembly, autonomous driving applications, augment reality and virtual reality (AR/VR), and medical applications. The code, the model weight, the deployment guide, and the evaluation protocol are: http://tianrun-chen.github.io/Reason3D/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
