PDMP：通过性能主导模态优先级重思平衡多模态学习
PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization
AI总结：
本文提出PDMP策略，通过性能主导模态优先级提升多模态学习效果，解决传统方法中因模态不平衡导致的优化不足问题。
摘要：
多模态学习因其实用性受到越来越多关注，但往往面临优化不足的问题，即多模态模型甚至不如单模态模型。现有方法将此问题归因于模态间的不平衡学习，并通过梯度调节解决。本文认为平衡学习并非多模态学习的最佳设置，相反，由性能主导模态驱动的不平衡学习能提升多模态性能。本文提出PDMP策略，通过挖掘独立训练的单模态模型的性能主导模态，并引入不对称系数调节各模态梯度，使性能主导模态主导优化过程。PDMP仅依赖单模态性能排名，不依赖多模态模型的结构和融合方法，具有广泛的应用潜力。最终，大量实验验证了PDMP的优越性。
英文摘要：
Multimodal learning has attracted increasing attention due to its practicality. However, it often suffers from insufficient optimization, where the multimodal model underperforms even compared to its unimodal counterparts. Existing methods attribute this problem to the imbalanced learning between modalities and solve it by gradient modulation. This paper argues that balanced learning is not the optimal setting for multimodal learning. On the contrary, imbalanced learning driven by the performance-dominant modality that has superior unimodal performance can contribute to better multimodal performance. And the under-optimization problem is caused by insufficient learning of the performance-dominant modality. To this end, we propose the Performance-Dominant Modality Prioritization (PDMP) strategy to assist multimodal learning. Specifically, PDMP firstly mines the performance-dominant modality via the performance ranking of the independently trained unimodal model. Then PDMP introduces asymmetric coefficients to modulate the gradients of each modality, enabling the performance-dominant modality to dominate the optimization. Since PDMP only relies on the unimodal performance ranking, it is independent of the structures and fusion methods of the multimodal model and has great potential for practical scenarios. Finally, extensive experiments on various datasets validate the superiority of PDMP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
