看见还是取悦：揭示视觉语言模型中的视觉谄媚与分裂信念
To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs
AI总结：
提出三层诊断框架，通过反事实干预实验发现视觉语言模型中普遍存在视觉谄媚（内部证据保留但输出幻觉答案）现象，并证明扩展模型规模无法解决该问题。
摘要：
当视觉语言模型正确回答时，它们是否真正依赖视觉信息？我们引入了一个三层诊断框架，包含三个每样本指标：潜在异常检测、视觉必要性分数和竞争分数，用于解耦感知、依赖和对齐失败。在9个视觉语言模型和9000个模型-样本对中，通过反事实盲、噪声和冲突干预，72.9%的样本表现出视觉谄媚，这是一种分裂信念模式，即内部证据被保留但解码出幻觉答案，而零样本表现出稳健拒绝，表明当前的对齐训练已消除拒绝作为解码结果。在Qwen-VL系列中，无论是代内还是代间扩展，都单调减少了语言捷径，但加剧了视觉谄媚，表明仅靠规模和更新的后训练无法解决接地问题。诊断分数进一步实现了一种无需训练的择性预测策略，在50%覆盖率下准确率提升高达9.5个百分点。
英文摘要：
When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
