arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01833cs.MM
收藏

多模态情感分析中缺失模态的解耦专家混合方法

Mixture of Disentangled Experts with Missing Modalities for Robust Multimodal Sentiment Analysis

School of Cyber Science and Technology, Beihang University(北京航空航天大学网络空间安全学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机学院);School of Information Science and Technology, Beijing University of Technology(北京工业大学信息学部)

Xiang Li, Xiaoming Zhang, Dezhuang Miao, Xianfu Cheng, Dawei Li, Honggui Han, Zhoujun Li

AI总结:

DERL通过解耦专家和多层次重建策略,提升多模态情感分析在缺失模态下的鲁棒性和表现

摘要:

多模态情感分析(MSA)整合多种模态来推断人类情感,但现实世界中的噪声往往导致缺失或损坏的数据。然而,现有特征解耦方法难以处理异构信息在不确定缺失情况下的内部变化,使得从退化模态中学习有效多模态表示变得困难。为了解决这个问题,我们提出了DERL,一种用于鲁棒MSA的解耦专家表示学习框架。具体而言,DERL采用混合专家来适应性地将多模态输入解耦为正交的私有和共享表示空间。进一步开发了多层次重建策略,以提供协同监督,增强所学表示的表达能力和鲁棒性。最后,解耦的特征作为具有不同作用的模态专家,生成重要性感知的融合结果。在两个MSA基准上的广泛实验表明,DERL在各种缺失模态条件下优于现有最先进方法。例如,我们的方法在MOSI上内模缺失情况下,准确率(Acc-2)提高了2.47%,平均绝对误差(MAE)提高了2.25%。

英文摘要:

Multimodal Sentiment Analysis (MSA) integrates multiple modalities to infer human sentiment, but real-world noise often leads to missing or corrupted data. However, existing feature-disentangled methods struggle to handle the internal variations of heterogeneous information under uncertain missingness, making it difficult to learn effective multimodal representations from degraded modalities. To address this issue, we propose DERL, a Disentangled Expert Representation Learning framework for robust MSA. Specifically, DERL employs hybrid experts to adaptively disentangle multimodal inputs into orthogonal private and shared representation spaces. A multi-level reconstruction strategy is further developed to provide collaborative supervision, enhancing both the expressiveness and robustness of the learned representations. Finally, the disentangled features act as modality experts with distinct roles to generate importance-aware fusion results. Extensive experiments on two MSA benchmarks demonstrate that DERL outperforms state-of-the-art methods under various missing-modality conditions. For instance, our method achieves improvements of 2.47% in Acc-2 and 2.25% in MAE on MOSI under intra-modal missingness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑