通过自适应掩码子网络学习再平衡多模态优化
Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks
AI总结:
针对多模态学习中的模态不平衡问题,提出基于重要性采样、考虑模态显著性的自适应掩码子网络方法AMSS及增强版AMSS+,经收敛分析和大量实验验证其性能优于现有方法。
摘要:
多模态学习旨在通过统一来自不同模态的模型来提升性能,但在真实数据中常面临“模态不平衡”问题,导致模型偏向主导模态而忽视其他模态,进而限制其整体效能。为解决这一挑战,核心思路是平衡各模态的优化以实现联合最优。现有方法通常采用模态级控制机制来调整每个模态参数的更新,但这种全局更新机制忽略了每个参数的不同重要性。受子网络优化的启发,我们探索了一种基于均匀采样的优化策略,发现它比全局更新更有效。基于这一发现,我们进一步提出了一种新的基于重要性采样的元素级联合优化方法,称为考虑模态显著性的自适应掩码子网络(Adaptively Mask Subnetworks Considering Modal Significance, AMSS)。具体而言,我们引入互信息率来确定模态显著性,并采用非均匀自适应采样从每个模态中选择前景子网络进行参数更新,从而再平衡多模态学习。此外,我们通过收敛分析证明了AMSS策略的可靠性。基于理论见解,我们使用无偏估计进一步增强了多模态掩码子网络策略,称为AMSS+。大量实验表明,我们的方法优于对比方法。
英文摘要:
Multi-modal learning aims to enhance performance by unifying models from various modalities but often faces the "modality imbalance" problem in real data, leading to a bias towards dominant modalities and neglecting others, thereby limiting its overall effectiveness. To address this challenge, the core idea is to balance the optimization of each modality to achieve a joint optimum. Existing approaches often employ a modal-level control mechanism for adjusting the update of each modal parameter. However, such a global-wise updating mechanism ignores the different importance of each parameter. Inspired by subnetwork optimization, we explore a uniform sampling-based optimization strategy and find it more effective than global-wise updating. According to the findings, we further propose a novel importance sampling-based, element-wise joint optimization method, called Adaptively Mask Subnetworks Considering Modal Significance(AMSS). Specifically, we incorporate mutual information rates to determine the modal significance and employ non-uniform adaptive sampling to select foreground subnetworks from each modality for parameter updates, thereby rebalancing multi-modal learning. Additionally, we demonstrate the reliability of the AMSS strategy through convergence analysis. Building upon theoretical insights, we further enhance the multi-modal mask subnetwork strategy using unbiased estimation, referred to as AMSS+. Extensive experiments reveal the superiority of our approach over comparison methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。