arXiv 2601.099542026-01-22cs.CV

视觉-语言模型的空间盲区

The Spatial Blindspot of Vision-Language Models

Cohere Labs Community（Cohere实验室社区）；Indian Institute of Science（印度科学学院）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；Imperial College London（帝国理工学院）；Eisai Inc.（卫材公司）；EleutherAI（EleutherAI组织）；Georgia Institute of Technology（佐治亚理工学院）

Nahid Alam, Leema Krishna Murali, Siddhant Bharadwaj, Patrick Liu, Timothy Chung, Drishti Sharma, Akshata A, Kranthi Kiran, Wesley Tam, Bala Krishna S Vegesna

备注：: Work done as part of the EleutherAI SOAR Program

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出通过改进图像编码器和2D位置编码来提升视觉-语言模型的空间推理能力，以解决其在空间关系捕捉上的不足。

摘要：

视觉-语言模型（VLMs）虽然发展迅速，但其捕捉空间关系的能力仍是一个盲区。当前VLMs通常采用对比语言-图像预训练（CLIP）风格的图像编码器。训练配方往往将图像扁平化为1D补丁序列，丢弃了空间推理所需的2D结构。我们认为，这种缺乏空间意识是VLM设计中的缺失维度，也是需要空间定位的应用（如机器人和具身AI）的瓶颈。为解决这一问题，我们研究了（i）采用替代目标训练的图像编码器，以及（ii）2D位置编码。我们的实验表明，这些架构选择可以在多个基准上提高空间推理能力。

英文摘要：

Vision-language models (VLMs) have advanced rapidly, but their ability to capture spatial relationships remains a blindspot. Current VLMs are typically built with contrastive language-image pretraining (CLIP) style image encoders. The training recipe often flattens images into 1D patch sequences, discarding the 2D structure necessary for spatial reasoning. We argue that this lack of spatial awareness is a missing dimension in VLM design and a bottleneck for applications requiring spatial grounding, such as robotics and embodied AI. To address this, we investigate (i) image encoders trained with alternative objectives and (ii) 2D positional encodings. Our experiments show that these architectural choices can lead to improved spatial reasoning on several benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。