超越视觉编码器：识别和缓解大视觉-语言模型中的空间偏差
Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Models
AI总结：
本文提出AGCI机制，通过动态注入全局视觉上下文缓解大视觉-语言模型中的空间偏差问题，提升模型的空间鲁棒性和下游任务表现。
摘要：
大视觉-语言模型（LVLMs）在广泛多模态任务中取得了显著成功，但其对空间变化的鲁棒性仍不够充分。在本工作中，我们系统研究了LVLMs的空间偏差，考察当相同的关键视觉信息在图像中不同位置时，模型如何响应。通过受控探测实验，我们发现当前LVLMs在这样的空间位移下往往会产生不一致的输出，揭示了其语义理解中的明显空间偏差。进一步分析表明，这种偏差并非来自视觉编码器，而是来自视觉编码器与大语言模型之间注意力机制的不匹配，这破坏了全局信息流。受这一见解的启发，我们提出了自适应全局上下文注入（AGCI），一种轻量级机制，能够动态地将共享的全局视觉上下文注入到每个图像令牌中。AGCI无需架构修改，通过增强图像令牌的语义可访问性来缓解空间偏差，同时保持模型的固有能力。广泛的实验表明，AGCI不仅增强了LVLMs的空间鲁棒性，还在各种下游任务和幻觉基准上取得了强劲的表现。
英文摘要：
Large Vision-Language Models (LVLMs) have achieved remarkable success across a wide range of multimodal tasks, yet their robustness to spatial variations remains insufficiently understood. In this work, we conduct a systematic study of the spatial bias of LVLMs, examining how models respond when identical key visual information is placed at different locations within an image. Through controlled probing experiments, we observe that current LVLMs often produce inconsistent outputs under such spatial shifts, revealing a clear spatial bias in their semantic understanding. Further analysis indicates that this bias does not stem from the vision encoder, but rather from a mismatch in attention mechanisms between the vision encoder and the large language model, which disrupts the global information flow. Motivated by this insight, we propose Adaptive Global Context Injection (AGCI), a lightweight mechanism that dynamically injects shared global visual context into each image token. AGCI works without architectural modifications, mitigating spatial bias by enhancing the semantic accessibility of image tokens while preserving the model's intrinsic capabilities. Extensive experiments demonstrate that AGCI not only enhances the spatial robustness of LVLMs, but also achieves strong performance on various downstream tasks and hallucination benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
