VLM3：视觉语言模型是原生3D学习者
VLM3: Vision Language Models Are Native 3D Learners
AI总结：
本文提出VLM3，通过焦距统一、文本像素参考和数据混合缩放，使标准视觉语言模型无需复杂架构或损失函数即可高效掌握多种3D任务。
摘要：
视觉语言模型（VLM）通过提示使统一模型能够解决各种视觉任务，在语义理解方面表现出色。然而，3D理解仍然很大程度上依赖于具有复杂任务特定设计的专家视觉模型。本文要提出的关键论点是，VLM是原生的3D学习者。我们深入的大规模研究表明：1）焦距统一，2）基于文本的像素参考，以及3）数据混合和缩放，是有效3D学习所需的一切。模型架构变化、大模型、大量数据增强以及包括回归公式在内的复杂损失（其中许多构成了专家视觉模型的基础）实际上并不是必要条件。因此，我们提出了VLM3，一种具有最简单设计的可扩展方法，使标准VLM能够掌握多样的3D任务。VLM3不仅大幅提升了VLM深度估计的准确性（0.84 -> 0.9），还实现了多样的3D任务，如像素对应、相机姿态估计和物体级3D理解，在保持标准架构和基于文本的训练的同时，匹配了专家视觉模型的准确性。我们相信VLM3为简单且可扩展的3D学习开辟了新的范式。
英文摘要：
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
