OpenVoxel: 一种无需训练的稀疏体素分组与标注算法用于开放词汇3D场景理解
OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding
AI总结：
OpenVoxel通过无需训练的多模态大语言模型实现稀疏体素的分组与标注，提升开放词汇3D场景理解的性能。
摘要：
我们提出了OpenVoxel，一种无需训练的算法，用于对稀疏体素进行分组和标注，以解决开放词汇3D场景理解任务。给定从3D场景多视角图像获得的稀疏体素栅格化（SVR）模型，我们的OpenVoxel能够生成有意义的组，描述场景中的不同物体。此外，通过利用强大的视觉语言模型（VLMs）和多模态大语言模型（MLLMs），我们的OpenVoxel成功构建了一个信息丰富的场景地图，通过为每个组标注，使进一步的3D场景理解任务如开放词汇分割（OVS）或指称表达分割（RES）成为可能。与以往方法不同，我们的方法无需训练，不引入CLIP/BERT文本编码器的嵌入。相反，我们直接使用MLLMs进行文本到文本搜索。通过广泛的实验，我们的方法在最近的研究中表现出色，特别是在复杂的指称表达分割（RES）任务中。代码将公开。
英文摘要：
We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
