G$^{2}$D：通过梯度引导蒸馏增强多模态学习
G$^{2}$D: Boosting Multimodal Learning with Gradient-Guided Distillation
AI总结：
针对多模态学习中的模态不平衡问题，提出G$^{2}$D知识蒸馏框架，通过融合单模态与多模态目标及动态SMP提升弱模态作用，并在分类和回归任务上取得优于现有方法的性能。
摘要：
多模态学习旨在利用来自不同数据模态的信息，以实现更全面的性能。然而，传统多模态模型常受到模态不平衡问题的困扰，即一个或少数模态主导模型优化，导致特征表示欠佳且弱模态未得到充分利用。为应对这一挑战，我们提出了梯度引导蒸馏（Gradient-Guided Distillation，G$^{2}$D），这是一个知识蒸馏框架，通过融合单模态与多模态目标的定制损失函数来优化多模态模型。G$^{2}$D还在学习过程中引入了动态序列模态优先级（sequential modality prioritization，SMP）技术，以确保每种模态都能主导学习过程，避免较强模态掩盖较弱模态的陷阱。我们在多个真实世界数据集上验证了G$^{2}$D，结果表明G$^{2}$D在训练期间增强了弱模态的重要性，并在分类和回归任务上优于最先进方法。我们的代码可在https://github.com/rAIson-Lab/G2D获取。
英文摘要：
Multimodal learning aims to leverage information from diverse data modalities to achieve more comprehensive performance. However, conventional multimodal models often suffer from modality imbalance, where one or a few modalities dominate model optimization, leading to suboptimal feature representation and underutilization of weak modalities. To address this challenge, we introduce Gradient-Guided Distillation (G$^{2}$D), a knowledge distillation framework that optimizes the multimodal model with a custom-built loss function that fuses both unimodal and multimodal objectives. G$^{2}$D further incorporates a dynamic sequential modality prioritization (SMP) technique in the learning process to ensure each modality leads the learning process, avoiding the pitfall of stronger modalities overshadowing weaker ones. We validate G$^{2}$D on multiple real-world datasets and show that G$^{2}$D amplifies the significance of weak modalities while training and outperforms state-of-the-art methods in classification and regression tasks. Our code is available at https://github.com/rAIson-Lab/G2D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
