AI 中文总结
该研究提出注意力头的角色断裂现象，构建无需微调的轻量级线性检测器，在6个VLMs和4个基准上平均AUROC达93.23，可检测VLM幻觉并支持干预操作。
AI 中文摘要
尽管视觉语言生成取得了显著进展，视觉语言模型（Vision-Language Models, VLMs）仍易产生幻觉，生成与输入图像不一致或无依据的内容。现有研究大多围绕单一幻觉模式设计检测或缓解方法，如视觉文本失衡，但真实的VLM幻觉由多种模式混合产生，因此绑定单一模式的信号在不同模型和任务间难以保持稳定。从统一的头层面视角出发，我们发现幻觉引发的变化表现为各注意力头忠实上下文行为的局部偏离，将此现象命名为角色断裂（Role-Break）。详细分析显示，这些偏离在注意力头、上下文源和偏离方向上呈系统组织，且一旦保留头身份，产生的信号具有线性可解读性。基于这些发现，我们构建了一个基于角色断裂的轻量级线性检测器，无需对VLM进行微调，其特征维度低于5000，在6个VLMs和4个基准上达到平均AUROC 93.23。小规模干预实验进一步表明，在判别设置中可直接对检测到的标记进行操作。
英文摘要
Despite remarkable progress in vision-language generation, Vision-Language Models (VLMs) remain prone to hallucinations, producing content that is inconsistent with or unsupported by the input image. Existing works largely design detection or mitigation methods around one specific hallucination pattern, such as visual-textual imbalance, but real VLM hallucinations arise from a mixture of multiple patterns, so signals bound to a single pattern struggle to remain stable across models and tasks. Under a unified head-level view, we find that hallucination-induced changes manifest as localized deviations from each head's faithful contextual behavior, a phenomenon we term Role-Break. Detailed analysis reveals that these deviations are systematically organized across attention heads, contextual sources, and deviation directions, and that the resulting signal is linearly readable once head identity is preserved. Based on these findings, we build a lightweight linear detector on top of Role-Break that requires no fine-tuning of the VLM, whose feature dimension stays below 5,000 and reaches an average AUROC of 93.23 across six VLMs and four benchmarks. A small-scale intervention experiment further shows that the detected tokens can be directly acted upon in the discriminative setting.
