GLIMPSE：面向大视觉语言模型的整体跨模态可解释性
GLIMPSE: Holistic Cross-Modal Explainability for Large Vision-Language Models
AI总结：
针对大视觉语言模型视觉注意力难以解读的问题，提出轻量模型无关框架GLIMPSE，融合多种技术生成跨模态推理热力图，在忠实度和人类注意力对齐上表现更优，还可用于模型诊断与透明性保障。
摘要：
近期的大视觉语言模型（LVLM）在视觉问答（VQA）任务中展现出更强的能力。然而，解读LVLM将视觉注意力投向何处仍是一项重大挑战，且这对理解模型行为至关重要。我们提出GLIMPSE（全称为Gradient-Layer Importance Mapping for Prompted Visual Saliency Explanation，即面向提示驱动视觉显著性解释的梯度-层重要性映射），这是一个轻量、模型无关的框架，可将LVLM的输出共同归因于支撑开放式生成的最相关视觉证据与文本信号。GLIMPSE融合梯度加权注意力、自适应层传播与相关性加权令牌聚合，生成用于解读跨模态推理的整体响应级热力图，在忠实度上优于现有方法，且在人类注意力对齐指标上刷新了当前最优水平。我们展示了一种分析方法，可挖掘LVLM跨模态归因的细粒度洞察、追踪推理动态、分析系统性偏差、诊断幻觉与偏见并保障透明度。
英文摘要：
Recent large vision-language models (LVLMs) have advanced capabilities in visual question answering (VQA). However, interpreting where LVLMs direct their visual attention remains a significant challenge, yet is essential for understanding model behavior. We introduce GLIMPSE (Gradient-Layer Importance Mapping for Prompted Visual Saliency Explanation), a lightweight, model-agnostic framework that jointly attributes LVLM outputs to the most relevant visual evidence and textual signals that support open-ended generation. GLIMPSE fuses gradient-weighted attention, adaptive layer propagation, and relevance-weighted token aggregation to produce holistic response-level heat maps for interpreting cross-modal reasoning, outperforming prior methods in faithfulness and pushing the state-of-the-art in human-attention alignment. We demonstrate an analytic approach to uncover fine-grained insights into LVLM cross-modal attribution, trace reasoning dynamics, analyze systematic misalignment, diagnose hallucination and bias, and ensure transparency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
