使用多模态大语言模型评估视频问答基准中的模态偏差
Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models
AI总结：
本研究引入模态重要性分数（MIS）并利用最先进的MLLMs评估视频问答基准中的单模态偏差，发现现有数据集存在模态不平衡且模型难以有效整合多模态信息，为构建模态平衡数据集提供了指导。
摘要：
多模态大语言模型（MLLMs）能够同时处理视觉、文本和听觉数据，捕捉补充人类分析的见解。然而，现有的视频问答（VidQA）基准和数据集通常偏向于单一模态，尽管其目标是要求整合多种模态的高级推理技能来回答问题。在本研究中，我们引入了模态重要性分数（MIS）来识别这种偏差。该分数旨在评估哪种模态嵌入了回答问题所需的信息。此外，我们提出了一种使用最先进的MLLMs来估计模态重要性的创新方法，该方法可作为人类对模态感知判断的代理。通过该MIS，我们证明了现有数据集中单模态偏差的存在以及真正多模态问题的稀缺性。我们进一步通过多项消融研究验证了模态重要性分数，以评估MLLMs在排列特征集上的性能。结果表明，由于现有数据集中的模态不平衡，当前模型未能有效整合信息。我们提出的基于MLLM的MIS可以指导整理模态平衡的数据集，从而推进多模态学习并增强MLLMs理解和利用跨模态协同关系的能力。
英文摘要：
Multimodal large language models (MLLMs) can simultaneously process visual, textual, and auditory data, capturing insights that complement human analysis. However, existing video question-answering (VidQA) benchmarks and datasets often exhibit a bias toward a single modality, despite the goal of requiring advanced reasoning skills that integrate diverse modalities to answer the queries. In this work, we introduce the modality importance score (MIS) to identify such bias. It is designed to assess which modality embeds the necessary information to answer the question. Additionally, we propose an innovative method using state-of-the-art MLLMs to estimate the modality importance, which can serve as a proxy for human judgments of modality perception. With this MIS, we demonstrate the presence of unimodal bias and the scarcity of genuinely multimodal questions in existing datasets. We further validate the modality importance score with multiple ablation studies to evaluate the performance of MLLMs on permuted feature sets. Our results indicate that current models do not effectively integrate information due to modality imbalance in existing datasets. Our proposed MLLM-derived MIS can guide the curation of modality-balanced datasets that advance multimodal learning and enhance MLLMs' capabilities to understand and utilize synergistic relations across modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
