利用多损失梯度调制改进多模态学习
Improving Multimodal Learning with Multi-Loss Gradient Modulation
AI总结：
本文提出一种多损失梯度调制方法，通过双向动态调整各模态学习速度并在收敛时消除平衡效应，有效解决了多模态学习中单一模态主导的问题，在多个音视频数据集上取得显著性能提升。
摘要：
从音频和视频等多种模态中学习，为利用互补信息、增强鲁棒性以及改善上下文理解和性能提供了机会。然而，组合这些模态带来了挑战，尤其是当模态在数据结构、预测贡献及其学习过程的复杂性上存在差异时。据观察，单一模态可能会主导学习过程，阻碍对其他模态信息的有效利用，导致模型性能次优。为解决这一问题，绝大多数先前的工作建议评估单模态贡献并动态调整训练以使其均衡。我们在前人工作的基础上进行了改进，引入了多损失目标并进一步优化了平衡过程，使其能够双向动态调整每个模态的学习速度（加速和减速），并在收敛时逐步消除平衡效应。我们在三个音视频数据集上取得了卓越结果：在CREMA-D上，使用ResNet骨干编码器的模型超越了此前最佳水平1.9%至12.4%，而Conformer骨干模型在不同融合方法下带来了2.8%至14.1%的提升。在AVE上，提升幅度为2.7%至7.7%，而在UCF101上，增益高达6.1%。
英文摘要：
Learning from multiple modalities, such as audio and video, offers opportunities for leveraging complementary information, enhancing robustness, and improving contextual understanding and performance. However, combining such modalities presents challenges, especially when modalities differ in data structure, predictive contribution, and the complexity of their learning processes. It has been observed that one modality can potentially dominate the learning process, hindering the effective utilization of information from other modalities and leading to sub-optimal model performance. To address this issue the vast majority of previous works suggest to assess the unimodal contributions and dynamically adjust the training to equalize them. We improve upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon convergence. We achieve superior results across three audio-video datasets: on CREMA-D, models with ResNet backbone encoders surpass the previous best by 1.9% to 12.4%, and Conformer backbone models deliver improvements ranging from 2.8% to 14.1% across different fusion methods. On AVE, improvements range from 2.7% to 7.7%, while on UCF101, gains reach up to 6.1%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
