为何大规模多模态语言模型难以确定物体方向
Why MLLMs Struggle to Determine Object Orientations
AI总结：
本文研究大规模多模态语言模型在处理图像中2D物体方向推理任务时的困难，通过实验发现方向信息可从编码器表示中恢复，挑战了视觉编码器限制的假设。
摘要：
大规模多模态语言模型（MLLMs）在需要推理图像中2D物体方向的任务上表现不佳，如前所述。Tong等人和Nichols等人假设这些失败源于视觉编码器，因为常用的编码器如CLIP和SigLIP是为图像-文本语义对齐而不是几何推理训练的。我们设计了一个受控的实证协议来测试这一假设，通过测量是否能从编码器表示中恢复旋转。具体而言，我们检查了LLaVA OneVision和Qwen2.5-VL-7B-Instruct模型的SigLIP和ViT特征，分别使用完整图像，并检查LLaVA 1.5和1.6中的CLIP表示，使用旋转前景补丁与自然背景图像。我们的假设是编码器嵌入中没有保留方向信息，我们通过训练线性回归器来预测从编码特征中获得物体方向。与假设相反，我们发现方向信息可以从编码器表示中恢复：简单的线性模型能准确预测物体方向。这反驳了MLLM方向失败源于视觉编码器的假设。在拒绝了接受的假设即MLLMs在2D方向任务上困难是因为视觉编码器限制后，我们仍不清楚为何它们失败。尽管全面解释超出了本文范围，我们显示虽然存在，方向信息分散在成千上万的特征中。这可能或可能不会是MLLMs无法利用可用方向信息的原因。
英文摘要：
Multimodal Large Language Models (MLLMs) struggle with tasks that require reasoning about 2D object orientation in images, as documented in prior work. Tong et al. and Nichols et al. hypothesize that these failures originate in the visual encoder, since commonly used encoders such as CLIP and SigLIP are trained for image-text semantic alignment rather than geometric reasoning. We design a controlled empirical protocol to test this claim by measuring whether rotations can be recovered from encoder representations. In particular, we examine SigLIP and ViT features from LLaVA OneVision and Qwen2.5-VL-7B-Instruct models, respectively, using full images, and examine CLIP representations in LLaVA 1.5 and 1.6 using rotated foreground patches against natural background images. Our null hypothesis is that orientation information is not preserved in the encoder embeddings and we test this by training linear regressors to predict object orientation from encoded features. Contrary to the hypothesis, we find that orientation information is recoverable from encoder representations: simple linear models accurately predict object orientations from embeddings. This contradicts the assumption that MLLM orientation failures originate in the visual encoder. Having rejected the accepted hypothesis that MLLMs struggle with 2D orientation tasks because of visual encoder limitations, we still don't know why they fail. Although a full explanation is beyond the scope of this paper, we show that although present, orientation information is spread diffusely across tens of thousands of features. This may or may not be while MLLMs fail to exploit the available orientation information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
