通过反向非对称融合缓解多模态学习中的强模态坍塌问题
Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion
- Imperial College London（帝国理工学院）
- California Polytechnic State University（加州州立理工大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对多模态学习中强模态坍塌导致模型难超单模态基准的问题，提出反向非对称融合（IAF）方法，在三类基准上验证其可保留主导模态性能且最高提升单模态基准8.25%
AI中文摘要：
融合多模态本应提升模型性能，但在MultiHuSE数据集上，早期融合、晚期融合及对称注意力融合常无法超越最优单模态基准（文本）。对对称注意力融合模型的通路隔离分析显示，某一设置下融合后文本通路的准确率从74.9%降至56.4%，表明融合过程中主导模态会被削弱，该现象被称为强模态坍塌，可解释部分多模态模型无法超越单模态基准的原因。本文提出反向非对称融合（IAF），避免强制跨模态互注意力，通过让主导模态不变通过融合层保留其完整性，弱模态则以主导模态为上下文锚点进行注意力计算；融合前采用模态感知知识蒸馏增强弱模态性能。在三个具有不同模态层级的基准上评估IAF：文本主导数据集MultiHuSE、UR-FUNNY及音频-视觉主导数据集MUStARD。通路隔离实验表明，IAF在所有测试配置下均能将主导模态的内部准确率保持在其单模态上限，而对称融合在MultiHuSE上会使主导模态准确率最高下降18.5%；IAF相比最强单模态基准最高提升8.25%。
英文摘要：
Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isolation of a symmetric attention fusion model reveals that the text-pathway accuracy drops from 74.9% to 56.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration. We term this strong-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines. We propose Inverted Asymmetric Fusion (IAF), which avoids forcing mutual attention across modalities. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor. Before fusion, weaker modalities are strengthened using Modality-Aware Knowledge Distillation. We evaluate IAF on three benchmarks with different modality hierarchies: text-dominant datasets (MultiHuSE, UR-FUNNY) and an audio-visual-dominant dataset (MUStARD). Pathway isolation shows that IAF preserves the dominant modality's internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18.5% on MultiHuSE. IAF improves over the strongest unimodal baseline by up to 8.25%.
