用加权班扎夫交互解释视觉-语言编码器中的相似性
Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions
AI总结：
针对现有视觉-语言编码器相似性解释仅捕获一阶归因、忽略跨模态交互的问题，提出基于加权Banzhaf交互指数的FIxLIP方法，实现更高效灵活的二阶交互解释，实验验证其性能优于一阶方法且可用于模型对比。
摘要：
语言-图像预训练（LIP）推动了具备零样本分类、定位、多模态检索与语义理解能力的视觉-语言模型发展。目前已有多种解释方法用于可视化输入图文对对模型相似性输出的重要性，但主流显著性图仅能捕获一阶归因，忽略了这类编码器内在的复杂跨模态交互。本文提出LIP模型可信交互解释方法FIxLIP，作为分解视觉-语言编码器相似性的统一方案。FIxLIP以博弈论为基础，分析表明相较于Shapley交互量化框架，加权Banzhaf交互指数具备更强灵活性与更高计算效率。实践层面，本文提出将指向博弈、插入/删除曲线间面积等解释评估指标自然拓展至二阶交互解释的方法。在MS COCO与ImageNet-1k基准上的实验验证，FIxLIP这类二阶方法性能优于一阶归因方法。除提供高质量解释外，本文还验证了FIxLIP可用于对比CLIP与SigLIP-2等不同模型的效用。
英文摘要：
Language-image pre-training (LIP) enables the development of vision-language models capable of zero-shot classification, localization, multimodal retrieval, and semantic understanding. Various explanation methods have been proposed to visualize the importance of input image-text pairs on the model's similarity outputs. However, popular saliency maps are limited by capturing only first-order attributions, overlooking the complex cross-modal interactions intrinsic to such encoders. We introduce faithful interaction explanations of LIP models (FIxLIP) as a unified approach to decomposing the similarity in vision-language encoders. FIxLIP is rooted in game theory, where we analyze how using the weighted Banzhaf interaction index offers greater flexibility and improves computational efficiency over the Shapley interaction quantification framework. From a practical perspective, we propose how to naturally extend explanation evaluation metrics, such as the pointing game and area between the insertion/deletion curves, to second-order interaction explanations. Experiments on the MS COCO and ImageNet-1k benchmarks validate that second-order methods, such as FIxLIP, outperform first-order attribution methods. Beyond delivering high-quality explanations, we demonstrate the utility of FIxLIP in comparing different models, e.g. CLIP vs. SigLIP-2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
