arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.13697cs.CVcs.AIcs.CL
收藏

利用多模态基础模型迈向鲁棒多模态学习

Toward Robust Multimodal Learning using Multimodal Foundational Models

Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳));Singapore University of Technology and Design(新加坡科技设计大学)

Xianbing Zhao, Soujanya Poria, Xuejiao Li, Yixin Chen, Buzhou Tang

AI总结:

针对多模态情感分析中模态随机缺失问题,本文提出 TRML 框架,通过生成虚拟模态并对齐其与缺失模态的语义空间,在三个基准数据集上验证了优越性能。

摘要:

现有的多模态情感分析任务高度依赖训练集和测试集均为完整多模态数据这一假设,然而该假设可能难以成立:在真实场景中,多模态数据往往是不完整的。因此,迫切需要一种在模态随机缺失场景下具有鲁棒性的多模态模型。近年来,基于 CLIP 的多模态基础模型通过学习图像与文本对的对齐跨模态语义,已在众多多模态任务上展现出令人瞩目的性能,但这些多模态基础模型同样无法直接应对涉及模态缺失的场景。为缓解这一问题,我们提出了一个简单且有效的框架,名为 TRML,即“利用多模态基础模型迈向鲁棒多模态学习”。TRML 采用生成的虚拟模态来替代缺失模态,并对齐生成模态与缺失模态之间的语义空间。具体而言,我们设计了一个缺失模态推断模块,用于生成虚拟模态并替代缺失模态。我们还设计了一个语义匹配学习模块,以对齐生成模态与缺失模态的语义空间。在完整模态的提示下,我们的模型通过利用已对齐的跨模态语义空间来捕获缺失模态的语义。实验在三个多模态情感分析基准数据集 CMU-MOSI、CMU-MOSEI 和 MELD 上证明了本方法的优越性。

英文摘要:

Existing multimodal sentiment analysis tasks are highly rely on the assumption that the training and test sets are complete multimodal data, while this assumption can be difficult to hold: the multimodal data are often incomplete in real-world scenarios. Therefore, a robust multimodal model in scenarios with randomly missing modalities is highly preferred. Recently, CLIP-based multimodal foundational models have demonstrated impressive performance on numerous multimodal tasks by learning the aligned cross-modal semantics of image and text pairs, but the multimodal foundational models are also unable to directly address scenarios involving modality absence. To alleviate this issue, we propose a simple and effective framework, namely TRML, Toward Robust Multimodal Learning using Multimodal Foundational Models. TRML employs generated virtual modalities to replace missing modalities, and aligns the semantic spaces between the generated and missing modalities. Concretely, we design a missing modality inference module to generate virtual modaliites and replace missing modalities. We also design a semantic matching learning module to align semantic spaces generated and missing modalities. Under the prompt of complete modality, our model captures the semantics of missing modalities by leveraging the aligned cross-modal semantic space. Experiments demonstrate the superiority of our approach on three multimodal sentiment analysis benchmark datasets, CMU-MOSI, CMU-MOSEI, and MELD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑