GaussianVLM：使用语言对齐高斯泼溅的以场景为中心的三维视觉-语言模型，用于具身推理及其他任务
GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond
AI总结：
本文提出GaussianVLM，通过将语言特征嵌入三维高斯泼溅场景并利用双重稀疏化器生成任务感知令牌，构建了首个基于高斯泼溅的三维视觉-语言模型，在域外设置中将先前三维VLM性能提升五倍。
摘要：
随着多模态语言模型的进步，其在三维场景理解中的应用是一个快速发展的前沿领域，推动着三维视觉-语言模型（VLMs）的发展。当前方法表现出对目标检测器的强依赖，引入了处理瓶颈和分类灵活性方面的限制。为了解决这些限制，我们提出了一种面向三维高斯泼溅场景的以场景为中心的三维VLM，该模型采用语言感知和任务感知的场景表示。我们的方法通过将语言与每个高斯基元相关联，直接将丰富的语言特征嵌入到三维场景表示中，实现了早期的模态对齐。为了处理由此产生的密集表示，我们引入了一个双重稀疏化器，通过任务引导和位置引导的路径将其蒸馏为紧凑的、与任务相关的令牌，从而生成稀疏的、任务感知的全局和局部场景令牌。值得注意的是，我们提出了第一个基于高斯泼溅的VLM，利用从标准RGB图像中获得的逼真三维表示，展示了强大的泛化能力：在域外设置中，它将先前三维VLM的性能提高了五倍。
英文摘要：
As multimodal language models advance, their application to 3D scene understanding is a fast-growing frontier, driving the development of 3D Vision-Language Models (VLMs). Current methods show strong dependence on object detectors, introducing processing bottlenecks and limitations in taxonomic flexibility. To address these limitations, we propose a scene-centric 3D VLM for 3D Gaussian splat scenes that employs language- and task-aware scene representations. Our approach directly embeds rich linguistic features into the 3D scene representation by associating language with each Gaussian primitive, achieving early modality alignment. To process the resulting dense representations, we introduce a dual sparsifier that distills them into compact, task-relevant tokens via task-guided and location-guided pathways, producing sparse, task-aware global and local scene tokens. Notably, we present the first Gaussian splatting-based VLM, leveraging photorealistic 3D representations derived from standard RGB images, demonstrating strong generalization: it improves performance of prior 3D VLM five folds, in out-of-the-domain settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
