面向模态不完整多模态情感分析的相关性解耦知识蒸馏
Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
AI总结：
针对多模态情感分析中模态不确定缺失问题，提出相关性解耦知识蒸馏框架 CorrKD，通过样本级对比蒸馏、类别引导原型蒸馏和响应解耦一致性蒸馏重建缺失语义并优化决策边界，在三个数据集上取得优于基线的性能。
摘要：
多模态情感分析（MSA）旨在通过多模态数据理解人类情感。大多数 MSA 工作基于模态完整性的假设。然而，在实际应用中，一些现实因素会导致不确定的模态缺失，从而严重降低模型性能。为此，我们提出了一种相关性解耦知识蒸馏（CorrKD）框架，用于处理不确定模态缺失下的 MSA 任务。具体而言，我们提出了一种样本级对比蒸馏机制，该机制传递包含跨样本相关性的综合知识，以重建缺失的语义。此外，引入了一种类别引导的原型蒸馏机制，利用类别原型捕获跨类别相关性，以对齐特征分布并生成有利的联合表示。最后，我们设计了一种响应解耦一致性蒸馏策略，通过响应解耦和互信息最大化来优化学生网络的情感决策边界。在三个数据集上的综合实验表明，与若干基线方法相比，我们的框架能够取得有利的提升。
英文摘要：
Multimodal sentiment analysis (MSA) aims to understand human sentiment through multimodal data. Most MSA efforts are based on the assumption of modality completeness. However, in real-world applications, some practical factors cause uncertain modality missingness, which drastically degrades the model's performance. To this end, we propose a Correlation-decoupled Knowledge Distillation (CorrKD) framework for the MSA task under uncertain missing modalities. Specifically, we present a sample-level contrastive distillation mechanism that transfers comprehensive knowledge containing cross-sample correlations to reconstruct missing semantics. Moreover, a category-guided prototype distillation mechanism is introduced to capture cross-category correlations using category prototypes to align feature distributions and generate favorable joint representations. Eventually, we design a response-disentangled consistency distillation strategy to optimize the sentiment decision boundaries of the student network through response disentanglement and mutual information maximization. Comprehensive experiments on three datasets indicate that our framework can achieve favorable improvements compared with several baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
