通过自蒸馏与统一模态交叉注意力增强模态缺失场景下的多模态情感分析
Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention
AI总结：
针对多模态情感分析中文本模态易缺失的问题，提出含UMCA和MIA的双流自蒸馏框架，结合RNC损失对齐模拟与真实文本表示，在文本缺失场景下的CMU-MOSEI数据集MAE指标上显著优于现有模型。
摘要：
在多模态情感分析中，由于标注成本更高且自动语音识别（ASR）质量不稳定，文本数据的收集通常比视频或音频数据更具挑战性。为应对这一难题，本研究构建了一个鲁棒模型，即便在文本模态缺失的情况下也能有效整合多模态情感信息。具体而言，我们提出了双流自蒸馏框架（Double-Flow Self-Distillation Framework），包含统一模态交叉注意力（Unified Modality Cross-Attention, UMCA）与模态想象自编码器（Modality Imagination Autoencoder, MIA），可出色处理模态完整及文本模态缺失两种场景。详细来说，当文本模态缺失时，框架利用基于大语言模型（LLM）的模型从音频模态模拟文本表示，同时MIA模块补充另外两种模态的信息，使模拟的文本表示与真实文本表示相近。为进一步对齐模拟表示与真实表示，并让模型捕捉情感效价回归任务中样本顺序的连续性，我们还引入了Rank-N对比（RNC）损失函数。在CMU-MOSEI数据集上测试时，我们的模型在平均绝对误差（MAE）指标上取得了优异表现，且在文本模态缺失时显著优于其他模型。代码可在以下地址获取：https://github.com/WarmCongee/SDUMC
英文摘要：
In multimodal sentiment analysis, collecting text data is often more challenging than video or audio due to higher annotation costs and inconsistent automatic speech recognition (ASR) quality. To address this challenge, our study has developed a robust model that effectively integrates multimodal sentiment information, even in the absence of text modality. Specifically, we have developed a Double-Flow Self-Distillation Framework, including Unified Modality Cross-Attention (UMCA) and Modality Imagination Autoencoder (MIA), which excels at processing both scenarios with complete modalities and those with missing text modality. In detail, when the text modality is missing, our framework uses the LLM-based model to simulate the text representation from the audio modality, while the MIA module supplements information from the other two modalities to make the simulated text representation similar to the real text representation. To further align the simulated and real representations, and to enable the model to capture the continuous nature of sample orders in sentiment valence regression tasks, we have also introduced the Rank-N Contrast (RNC) loss function. When testing on the CMU-MOSEI, our model achieved outstanding performance on MAE and significantly outperformed other models when text modality is missing. The code is available at: https://github.com/WarmCongee/SDUMC
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
