你的 3D 编码器真的有效吗？当来自 2D VLM 的预训练-SFT 遇上 3D VLM
Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs
AI总结：
本文分析以 3D 场景为中心的 VLM 性能受限原因，发现其弱用 3D 编码器、依赖语言捷径，并提出 3D Relevance Discrimination QA 数据集以促进真实 3D 理解。
摘要：
2D 视觉-语言模型（Vision-Language Models，VLM）的显著进展，激发了人们将其扩展到 3D 场景以完成 3D 问答、密集描述和视觉定位等任务的兴趣。不同于通常通过图像编码器处理图像的 2D VLM，具有复杂空间结构的 3D 场景允许多样化的模型架构。根据编码器设计，本文将近期 3D VLM 分为以 3D 对象为中心、基于 2D 图像和以 3D 场景为中心的方法。尽管以 3D 场景为中心的 VLM 在架构上与对应的 2D 模型相似，但与最新的以 3D 对象为中心和基于 2D 图像的方法相比，它们表现出相对较低的性能。为理解这一差距，我们进行了深入分析，结果表明以 3D 场景为中心的 VLM 对 3D 场景编码器的依赖有限，并且预训练阶段的效果似乎不如 2D VLM 中的预训练。此外，我们观察到，在更大的数据集上，数据规模扩展带来的收益并不明显。研究表明，尽管这些模型具备跨模态对齐能力，但它们倾向于过度依赖语言线索，并对高频答案分布过拟合，从而降低了对 3D 编码器的有效利用。为解决这些局限并促进真正的 3D 场景理解，我们引入了一个新的 3D Relevance Discrimination QA 数据集，旨在破坏捷径学习并提升 3D 理解。我们的发现强调，需要更先进的评估和改进策略，以在 3D VLM 中实现更好的 3D 理解。
英文摘要：
Remarkable progress in 2D Vision-Language Models (VLMs) has spurred interest in extending them to 3D settings for tasks like 3D Question Answering, Dense Captioning, and Visual Grounding. Unlike 2D VLMs that typically process images through an image encoder, 3D scenes, with their intricate spatial structures, allow for diverse model architectures. Based on their encoder design, this paper categorizes recent 3D VLMs into 3D object-centric, 2D image-based, and 3D scene-centric approaches. Despite the architectural similarity of 3D scene-centric VLMs to their 2D counterparts, they have exhibited comparatively lower performance compared with the latest 3D object-centric and 2D image-based approaches. To understand this gap, we conduct an in-depth analysis, revealing that 3D scene-centric VLMs show limited reliance on the 3D scene encoder, and the pre-train stage appears less effective than in 2D VLMs. Furthermore, we observe that data scaling benefits are less pronounced on larger datasets. Our investigation suggests that while these models possess cross-modal alignment capabilities, they tend to over-rely on linguistic cues and overfit to frequent answer distributions, thereby diminishing the effective utilization of the 3D encoder. To address these limitations and encourage genuine 3D scene understanding, we introduce a novel 3D Relevance Discrimination QA dataset designed to disrupt shortcut learning and improve 3D understanding. Our findings highlight the need for advanced evaluation and improved strategies for better 3D understanding in 3D VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
