MM-SHAP：一种用于衡量视觉与语言模型及任务中多模态贡献的性能无关指标
MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks
AI总结：
提出MM-SHAP，一种基于Shapley值的性能无关多模态贡献度量，用于量化视觉语言模型中各模态的使用比例，实验发现单模态坍缩可多方向发生，推荐用于多模态任务分析。
摘要：
视觉与语言模型（VL）已知会利用单个模态中的不稳健指标（例如由分布偏差引入的指标），而不是关注每个模态中的相关信息。当单模态模型在视觉语言任务上达到与多模态模型相似的准确率时，表明发生了所谓的单模态坍缩。然而，基于准确率的测试无法检测例如模型预测错误但模型使用了来自某一模态的相关信息的情况。为此，我们提出了MM-SHAP，一种基于Shapley值的性能无关的多模态得分，能够可靠地量化多模态模型使用各单一模态的比例。我们以两种方式应用MM-SHAP：（1）比较模型在平均多模态程度上的差异；（2）衡量单个模型在不同任务和数据集上各单一模态的贡献。在四个视觉语言任务上对六个视觉语言模型（LXMERT、CLIP和四个ALBEF变体）进行的实验表明，单模态坍缩可能以不同程度和不同方向发生，这与单模态坍缩是单向的普遍假设相矛盾。基于我们的结果，我们推荐使用MM-SHAP来分析多模态任务，以诊断并引导多模态集成的发展。代码可在\url{https://github.com/Heidelberg-NLP/MM-SHAP}获取。
英文摘要：
Vision and language models (VL) are known to exploit unrobust indicators in individual modalities (e.g., introduced by distributional biases) instead of focusing on relevant information in each modality. That a unimodal model achieves similar accuracy on a VL task to a multimodal one, indicates that so-called unimodal collapse occurred. However, accuracy-based tests fail to detect e.g., when the model prediction is wrong, while the model used relevant information from a modality. Instead, we propose MM-SHAP, a performance-agnostic multimodality score based on Shapley values that reliably quantifies in which proportions a multimodal model uses individual modalities. We apply MM-SHAP in two ways: (1) to compare models for their average degree of multimodality, and (2) to measure for individual models the contribution of individual modalities for different tasks and datasets. Experiments with six VL models -- LXMERT, CLIP and four ALBEF variants -- on four VL tasks highlight that unimodal collapse can occur to different degrees and in different directions, contradicting the wide-spread assumption that unimodal collapse is one-sided. Based on our results, we recommend MM-SHAP for analysing multimodal tasks, to diagnose and guide progress towards multimodal integration. Code available at \url{https://github.com/Heidelberg-NLP/MM-SHAP}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
