通过反事实语义显著性揭示人类与VLM场景感知之间的差距
Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
AI总结：
本文通过反事实语义显著性方法揭示VLM在高阶语义场景理解中与人类感知之间的差距，发现模型过度依赖大物体、图像中心物体和高显著性物体，而对人物依赖程度较低。
摘要：
评估大型视觉-语言模型（VLMs）是否与人类感知在高阶语义场景理解上一致仍是一个挑战。传统白盒可解释性方法无法应用于闭源架构，被动指标无法隔离因果特征。我们引入反事实语义显著性（CSS）。这是一个黑盒、模型无关的框架，通过测量物体因果消融导致的语义变化来量化其重要性。为了评估AI-人类语义对齐，我们测试了 prominent VLMs 对比包含16,289个有效响应的16,289个有效响应的307个复杂自然场景和1,306个高保真反事实变体的人类心理物理学基准。我们的分析揭示了一个普遍的场景理解差距：模型相对于人类表现出对大物体（大小偏见）、图像中心物体（中心偏见）和高显著性物体的过度依赖。相反，模型对场景中人物的依赖程度低于人类参与者。模型的大小偏见是解释模型-人类语义分歧变化的主要驱动因素。代码和数据将在https://github.com/starsky77/Counterfactual-Semantic-Saliency上提供。
英文摘要：
Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpretability methods are inapplicable to closed-source architectures and passive metrics fail to isolate causal features. We introduce Counterfactual Semantic Saliency (CSS). This black-box, model-agnostic framework quantifies the importance of objects by measuring the semantic shift induced by their causal ablation from a scene. To evaluate AI-human semantic alignment, we tested prominent VLMs against a human psychophysics baseline comprising 16,289 valid responses across 307 complex natural scenes and 1,306 high-fidelity counterfactual variants. Our analysis reveals a pervasive scene comprehension gap: models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias), and high saliency objects. In contrast, models rely less on people in the scenes than our human participants to describe the images. A model's size bias is a primary driver explaining variations in model-human semantic divergence. Code and data will be available at https://github.com/starsky77/Counterfactual-Semantic-Saliency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
