语言覆盖视觉：视觉语言模型中的过度对齐与几何去偏
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
AI总结：
本文研究视觉语言模型中过度对齐导致的幻觉问题，提出几何去偏方法，通过投影消除文本子空间影响，减少幻觉并提升长文本生成性能。
摘要：
视觉语言模型（VLMs）日益用于高风险应用，如医疗成像和自动驾驶，但它们经常产生幻觉，自信描述不存在于输入中的内容。本文通过机械分析研究这些失败模式的根源，聚焦于基于解码器的VLMs。我们发现这些失败模式源于几何过度对齐：为弥合模态差距，解码器VLMs将视觉嵌入过度对齐到文本流形，注入统计语言偏见，系统性地掩盖细粒度视觉证据。尽管先前工作要么激烈关闭这一差距，要么通过昂贵的黑箱解码策略压制幻觉，但没有解决根本的几何原因。我们提供了这种过度对齐的首次定量表征，证明语言偏见集中在通用、跨数据集的文本子空间的前几个主成分中。基于这一见解，我们提出两种互补的解决方案：一种无需训练的推理策略和一种偏见感知的微调范式，两者均显式地从视觉表示中投影出该子空间。我们的方法在POPE、CHAIR和AMBER基准上显著减少幻觉，并在长文本生成任务中提高CLAIR分数，其中无需训练的变体在基础模型上无计算开销。
英文摘要：
Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input. We investigate the root causes of these failure modes with a mechanistic analysis focusing on the decoder-based VLMs. We trace these failure modes to a geometric over-alignment: to bridge the modality gap required by attention mechanisms, decoder-based VLMs over-align visual embeddings with the text manifold, injecting a statistical linguistic bias that systematically overshadows fine-grained visual evidence. While prior work either aggressively closes this gap or suppresses hallucinations through expensive black-box decoding strategies, none addresses the underlying geometric cause. We provide the first quantitative characterization of this over-alignment, demonstrating that linguistic bias concentrates in the top principal components of a universal, dataset-agnostic text subspace. Building on this insight, we propose two complementary remedies: a training-free inference strategy and a bias-aware fine-tuning paradigm, both of which explicitly project out this subspace from visual representations. Our methods significantly reduce hallucinations across POPE, CHAIR, and AMBER benchmarks, and improve CLAIR scores on long-form captioning tasks, with the training-free variant adding no computational overhead over the base model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
