LLaVA$^3$：像立体主义画家一样表示3D场景以提升VLM的3D场景理解
LLaVA$^3$: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs
AI总结：
LLaVA$^3$通过立体主义方法提升VLM对3D场景的理解能力，利用多视角2D图像无需微调实现更优的3D场景理解。
摘要：
开发能够理解3D场景的多模态语言模型仍然具有挑战性，因为3D训练数据有限，而用于视觉语言模型（VLM）的2D数据集却非常丰富。作为替代方案，我们引入了LLaVA$^3$（发音为LLaVA-Cube），一种新的方法，通过仅使用多视角2D图像和不进行微调来提高VLM对3D场景的理解能力。受立体派画家的启发，他们在一个画面中表示了3D物体的多个视角，我们提出通过每个物体的全方位视觉表示来描述3D场景给VLM。这些表示是从场景的中间多视角3D重建中获得的。在3D VQA和3D语言定位的大量实验中，我们的方法优于先前的2D基于VLM解决方案。
英文摘要：
Developing a multi-modal language model capable of understanding 3D scenes remains challenging due to the limited availability of 3D training data, in contrast to the abundance of 2D datasets used for vision-language models (VLM). As an alternative, we introduce LLaVA$^3$ (pronounced LLaVA-Cube), a novel method that improves the 3D scene understanding capabilities of VLM using only multi-view 2D images and without any fine-tuning. Inspired by Cubist painters, who represented multiple viewpoints of a 3D object within a single picture, we propose to describe the 3D scene for the VLM through omnidirectional visual representations of each object. These representations are derived from an intermediate multi-view 3D reconstruction of the scene. Extensive experiments on 3D VQA and 3D language grounding show that our approach outperforms previous 2D-based VLM solutions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
