arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00811cs.AI
收藏

MissMAC-Bench: 构建缺失模态问题的坚实基准以提升鲁棒多模态情感计算

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

Sun Yat-Sen University(中山大学);South China Normal University(华南师范大学);Pazhou Laboratory(琶洲实验室)

Ronghao Lin, Honghao Lu, Ruixing Wu, Aolin Xiong, Qinggong Chu, Qiaolin He, Sijie Mai, Haifeng Hu

AI总结:

MissMAC-Bench通过构建统一评估标准和跨模态协同原则,系统量化缺失模态问题,提升多模态情感计算的鲁棒性和实际应用能力。

摘要:

作为一种跨异构数据源的知识发现任务,当前多模态情感计算(MAC)严重依赖多种模态的完整性以准确理解人类的情感状态。然而,在现实场景中,模态数据的可用性往往是动态且不确定的,导致由于分布偏移和语义缺陷的不完整多模态输入而产生显著的性能波动。被称为缺失模态问题,这一挑战对MAC模型的鲁棒性和实际部署构成了关键障碍。为了系统量化这一问题,我们引入MissMAC-Bench,一个全面的基准,旨在从跨模态协同的角度建立公平且统一的评估标准。提出了两个指导原则,包括训练过程中不缺失先验知识,以及一个能够处理完整和不完整模态场景的单一模型,从而确保更好的泛化能力。此外,为了弥合学术研究与现实应用之间的差距,我们的基准集成了评估协议,同时在数据集和实例层面整合了固定和随机缺失模式。在四个数据集上对三种广泛使用的语言模型进行的广泛实验验证了各种MAC方法在解决缺失模态问题上的有效性。我们的基准为推进鲁棒多模态情感计算提供了坚实的基础,并促进了多媒体数据挖掘的发展。

英文摘要:

Current Multimodal Affective Computing (MAC) systems heavily rely on the completeness of multiple modalities to accurately understand human's affective state. However, in real-world scenarios, the availability of modality data is often dynamic and uncertain, leading to substantial performance fluctuations due to the distribution shifts and semantic deficiencies of the incomplete multimodal inputs. Known as the missing modality issue, this challenge poses a critical barrier to the robustness and practical deployment of MAC models. To systematically quantify this issue, we introduce \textbf{MissMAC-Bench}, a comprehensive benchmark designed to establish fair and unified evaluation standards from the perspective of cross-modal synergy. Two guiding principles are proposed, including no missing prior during training, and one single model capable of handling both complete and incomplete modality scenarios, thereby ensuring better generalization. Moreover, to bridge the gap between academic research and real-world applications, our benchmark integrates evaluation protocols with both fixed and random missing patterns at the dataset and instance levels. Extensive experiments conducted on 3 widely-used language models across 4 datasets validate the effectiveness of diverse MAC approaches in tackling the missing modality issue. Our benchmark provides a solid foundation for advancing robust MAC and promotes the development of multimedia data mining. Our code is released in https://github.com/RH-Lin/MissMAC-Bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑