DepthLM：基于视觉语言模型的度量深度估计
DepthLM: Metric Depth From Vision Language Models
AI总结：
针对视觉语言模型3D理解能力不足的问题，提出DepthLM方法，通过带稀疏标签的文本监督微调、视觉提示和内参条件增强，让VLM在度量深度估计上首次媲美纯视觉模型，还可拓展至更多3D任务。
摘要：
视觉语言模型（VLM）可通过文本交互灵活处理各类视觉任务。尽管当前最先进的VLM（包括GPT-5）在语义理解方面表现出色，但在从2D输入理解3D信息方面仍存在困难。另一方面，专业的纯视觉模型在度量深度估计这一关键3D理解任务上能达到超越人类的精度，但它们需要任务特定的架构和损失函数。这种差异促使我们提出问题：在不改变架构或损失函数的前提下，VLM能否达到专家级精度？我们以逐像素度量深度估计为代表性任务，证明答案是肯定的！令人惊讶的是，综合分析表明，带有稀疏标签的基于文本的监督微调足以让VLM解锁强大的3D理解能力，无需密集预测头或复杂的回归/正则化损失。VLM的瓶颈实际上在于像素参考和跨数据集相机歧义，我们通过视觉提示和内参条件增强来解决这些问题。使用小得多的模型，我们的方法DepthLM的精度超过了最先进的VLM两倍以上，首次让VLM可与纯视觉模型相媲美。有趣的是，尽管训练期间没有显式约束，经DepthLM训练的VLM自然避免了过度平滑，边界区域的飞点数量远少于纯视觉模型。DepthLM的简洁性还使单个VLM能够覆盖度量深度之外的各种3D任务。我们的代码和模型将在以下链接发布。
英文摘要：
Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy in metric depth estimation, a key 3D understanding task. However, they require task-specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture or loss change? We take per-pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised-finetuning with sparse labels is sufficient for VLMs to unlock strong 3D understanding, no dense prediction head or complex regression/regularization loss is needed. The bottleneck for VLMs lies actually in pixel reference and cross-dataset camera ambiguity, which we address through visual prompting and intrinsic-conditioned augmentation. With much smaller models, our method DepthLM surpasses the accuracy of most advanced VLMs by over 2x, making VLMs for the first time comparable with pure vision models. Interestingly, without explicit enforcement during training, VLMs trained with DepthLM naturally avoids over-smoothing, having much fewer flying points at boundary regions than pure vision models. The simplicity of DepthLM also enables a single VLM to cover various 3D tasks beyond metric depth. Our code and model will be released at the link below.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
