用于鲁棒多模态情感分析的对比知识蒸馏
Contrastive Knowledge Distillation for Robust Multimodal Sentiment Analysis
AI总结:
本文提出MM-CKD方法,通过多视图监督对比学习转移知识,以较低计算成本解决多模态情感分析中模态不完整问题,提升师生模型性能。
摘要:
多模态情感分析(MSA)系统利用不同模态的信息来预测人类情感强度。模态不完整是可能导致MSA系统性能显著下降的重要问题。通过生成式插补,即从可用数据中恢复缺失数据,系统可能获得鲁棒的性能,但会导致高昂的计算成本。本文提出一种知识蒸馏方法,称为“多模态对比知识蒸馏”(MM-CKD),以较低的计算成本解决视频情感分析中模态不完整的问题,作为一种新颖的非插补方法。我们采用多视图监督对比学习(MVSC)将知识从教师模型转移到学生模型。该方法不仅利用跨模态知识,还引入了带监督的跨样本知识,通过在线学习共同提升教师和学生模型的性能。我们的方法取得了与最先进的基于插补的方法相当的结果,且计算成本显著更低。
英文摘要:
Multimodal sentiment analysis (MSA) systems leverage information from different modalities to predict human sentiment intensities. Incomplete modality is an important issue that may cause a significant performance drop in MSA systems. By generative imputation, i.e., recovering the missing data from available data, systems may achieve robust performance but will lead to high computational costs. This paper introduces a knowledge distillation method, called `Multi-Modal Contrastive Knowledge Distillation' (MM-CKD), to address the issue of incomplete modality in video sentiment analysis with lower computation cost, as a novel non-imputation-based method. We employ Multi-view Supervised Contrastive Learning (MVSC) to transfer knowledge from a teacher model to student models. This approach not only leverages cross-modal knowledge but also introduces cross-sample knowledge with supervision, jointly improving the performance of both teacher and student models through online learning. Our method gives competitive results with significantly lower computational costs than state-of-the-art imputation-based methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。