MILES：用于平衡多模态学习的模态感知学习率调度器
MILES: Modality-Informed Learning Rate Scheduler for Balancing Multimodal Learning
AI总结：
本文提出 MILES 学习率调度器，通过动态利用各模态条件利用率差异平衡训练，在四个联合融合任务上优于七个基线，并增强单模态与缺失模态场景下的编码器能力。
摘要：
多模态神经网络旨在结合被称为模态的多种数据源，以获得相较于依赖单一模态时更优的性能。然而，多模态网络的训练通常受到模态过拟合的阻碍，即网络过度依赖可用模态中的某一种。这往往会导致次优性能，限制多模态学习的潜力，并使其相对于单模态模型只能取得边际提升。在本工作中，我们提出了 Modality-Informed Learning ratE Scheduler（MILES，模态感知学习率调度器），用于以平衡的方式训练多模态联合融合模型。MILES 利用训练过程中各模态条件利用率之间的差异，来有效平衡多模态学习。学习率会在训练期间动态调整，以平衡多模态模型从每个模态学习的速度，目标是提升多模态预测和单模态预测两方面的性能。我们在四个多模态联合融合任务上对 MILES 进行了广泛评估，并将其性能与七个最先进基线进行比较。结果表明，在研究所涵盖的所有任务和融合方法上，MILES 均优于所有基线，并在训练过程中有效平衡了模态使用。这带来了更好的多模态性能和更强的模态编码器；在处理单模态样本或模态缺失时，可以利用这些更强的编码器。总体而言，我们的工作强调了平衡多模态学习对提升模型性能的影响。
英文摘要：
The aim of multimodal neural networks is to combine diverse data sources, referred to as modalities, to achieve enhanced performance compared to relying on a single modality. However, training of multimodal networks is typically hindered by modality overfitting, where the network relies excessively on one of the available modalities. This often yields sub-optimal performance, hindering the potential of multimodal learning and resulting in marginal improvements relative to unimodal models. In this work, we present the Modality-Informed Learning ratE Scheduler (MILES) for training multimodal joint fusion models in a balanced manner. MILES leverages the differences in modality-wise conditional utilization rates during training to effectively balance multimodal learning. The learning rate is dynamically adjusted during training to balance the speed of learning from each modality by the multimodal model, aiming for enhanced performance in both multimodal and unimodal predictions. We extensively evaluate MILES on four multimodal joint fusion tasks and compare its performance to seven state-of-the-art baselines. Our results show that MILES outperforms all baselines across all tasks and fusion methods considered in our study, effectively balancing modality usage during training. This results in improved multimodal performance and stronger modality encoders, which can be leveraged when dealing with unimodal samples or absent modalities. Overall, our work highlights the impact of balancing multimodal learning on improving model performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
