衡量跨模态协同：VLM可解释性的一个基准
Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
AI总结：
本文提出Synergistic Faithfulness作为衡量VLM跨模态协同的指标，解决了传统单模态评估方法在评估VLM可解释性时的不足，通过引入Shapley交互指数，实现了对多模态协同的准确评估，同时提升了计算效率。
摘要：
视觉-语言模型（VLMs）将复杂的视觉输入映射到语义空间，但目前解释VLM的跨模态推理仍依赖于通过单模态扰动度量评估的后验解释器。我们揭示了这一范式的局限性：由于多模态数据集包含语言先验和模态偏差，VLMs经常表现出跨模态冗余，允许它们仅使用文本回答视觉查询。因此，单模态度量惩罚忠实的解释器，导致评估崩溃，其中视觉和文本排名根本矛盾（Kendall's τ= -0.06）。为了解决这一问题，我们引入了Synergistic Faithfulness（F_syn），一个基于Shapley交互指数的可扩展度量，严格隔离模态间的Harsanyi收益，作为高度准确的替代指标（ρ= 0.92），同时实现了24倍的计算加速。在评估8种不同的XAI方法、3种VLM架构和3个基准数据集时，发现为VLM设计的解释器严重过度索引视觉显著性，并在捕捉真正的跨模态协同方面显著劣于适应的注意力方法。通过将视觉合理性与跨模态忠实性解耦，本文提供了一个严格评估框架，以安全审计VLM在高风险部署中的推理。
英文摘要：
Vision-Language Models (VLMs) map complex visual inputs to semantic spaces, but interpreting the cross-modal reasoning of VLMs currently relies on post-hoc explainers evaluated via unimodal perturbation metrics. We expose a limitation in this paradigm: because multimodal datasets contain language priors and modality biases, VLMs frequently exhibit cross-modal redundancy, allowing them to answer visual queries using text alone. Consequently, unimodal metrics penalize faithful explainers, triggering an evaluation collapse where visual and textual rankings fundamentally contradict each other. %(Kendall's $τ= -0.06$). To resolve this, we introduce Synergistic Faithfulness ($\mathcal{F}_{syn}$), a scalable metric rooted in the Shapley Interaction Index that strictly isolates the joint Harsanyi dividend between modalities, serving as a highly accurate surrogate ($ρ= 0.92$) while achieving a $24\times$ computational speedup. Evaluating 8 distinct XAI methods across 3 VLM architectures and 3 benchmark datasets, reveals that explainers proposed for VLMs heavily over-index on visual salience and significantly underperform adapted attention-based methods in capturing true cross-modal synergy. By decoupling visual plausibility from cross-modal faithfulness, this work provides a rigorous evaluation framework required to safely audit VLM reasoning in high-stakes deployments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
