SplatTalk：基于高斯泼溅的3D视觉问答
SplatTalk: 3D VQA with Gaussian Splatting
AI总结：
SplatTalk利用可泛化3D高斯泼溅生成3D标记，输入预训练LLM，实现仅凭带姿态图像的零样本3D视觉问答，性能优于专门训练的3D模型和2D-LMM基线，与最先进3D LMM相当。
摘要：
语言引导的3D场景理解对于推进机器人、AR/VR和人机交互应用至关重要，使模型能够通过自然语言理解和交互3D环境。尽管2D视觉语言模型（VLMs）在2D视觉问答任务中取得了显著成功，但由于3D数据的复杂性和手动标注的高成本，3D领域进展明显较慢。在这项工作中，我们引入了SplatTalk，一种新颖的方法，使用可泛化的3D高斯泼溅（3DGS）框架生成适合直接输入预训练大语言模型（LLM）的3D标记，从而仅凭带姿态的图像即可实现有效的零样本3D视觉问答（3D VQA）。在多个基准上的实验中，我们的方法优于专门为该任务训练的3D模型以及仅使用图像的先前基于2D-LMM的模型（我们的设置），同时与额外利用3D输入的最先进的3D LMM相比，取得了具有竞争力的性能。项目网站：https://splat-talk.github.io/
英文摘要：
Language-guided 3D scene understanding is important for advancing applications in robotics, AR/VR, and human-computer interaction, enabling models to comprehend and interact with 3D environments through natural language. While 2D vision-language models (VLMs) have achieved remarkable success in 2D VQA tasks, progress in the 3D domain has been significantly slower due to the complexity of 3D data and the high cost of manual annotations. In this work, we introduce SplatTalk, a novel method that uses a generalizable 3D Gaussian Splatting (3DGS) framework to produce 3D tokens suitable for direct input into a pretrained LLM, enabling effective zero-shot 3D visual question answering (3D VQA) for scenes with only posed images. During experiments on multiple benchmarks, our approach outperforms both 3D models trained specifically for the task and previous 2D-LMM-based models utilizing only images (our setting), while achieving competitive performance with state-of-the-art 3D LMMs that additionally utilize 3D inputs. Project website: https://splat-talk.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
