相同答案，不同表示：VLMs中的隐藏不稳定性
Same Answer, Different Representations: Hidden instability in VLMs
AI总结：
本研究揭示了视觉语言模型中隐藏的不稳定性，通过引入新的评估框架发现模型在内部表示漂移、鲁棒性与决策边界等方面存在显著问题。
摘要：
视觉语言模型（VLMs）的鲁棒性通常通过输出层面的不变性来评估，隐含假设稳定的预测反映稳定的多模态处理。在本工作中，我们认为这一假设是不足的。我们引入了一个具有表示意识和频率意识的评估框架，该框架测量内部嵌入漂移、频谱敏感性和结构平滑性（视觉标记的空间一致性），并结合标准标签基于指标。将此框架应用于SEEDBench、MMMU和POPE数据集上的现代VLMs，揭示了三种不同的失败模式。首先，模型在经历显著的内部表示漂移时，经常保留预测答案；对于文本叠加等扰动，这种漂移接近图像间变异性，表明表示移动到通常由无关输入占据的区域，尽管输出未变。其次，鲁棒性不随规模提高而改善；更大模型获得更高的准确性，但表现出相等或更高的敏感性，这与更尖锐但更脆弱的决策边界一致。第三，我们发现扰动对任务影响不同：当扰动破坏模型如何结合粗略和精细的视觉线索时，会损害推理，但在幻觉基准上，扰动可以减少假阳性，使模型生成更保守的答案。
英文摘要：
The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial internal representation drift; for perturbations such as text overlays, this drift approaches the magnitude of inter-image variability, indicating that representations move to regions typically occupied by unrelated inputs despite unchanged outputs. Second, robustness does not improve with scale; larger models achieve higher accuracy but exhibit equal or greater sensitivity, consistent with sharper yet more fragile decision boundaries. Third, we find that perturbations affect tasks differently: they harm reasoning when they disrupt how models combine coarse and fine visual cues, but on the hallucination benchmarks, they can reduce false positives by making models generate more conservative answers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
