发表机构
Beihang University; Institute of Computing Technology, Chinese Academy of Sciences; Amap, Alibaba Group; Kuaishou Technology; Beijing Jiaotong University; Renmin University of China（北京航空航天大学; 中国科学院计算技术研究所; 阿里巴巴集团高德; 快手科技; 北京交通大学; 中国人民大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对多模态学习中模态主导导致弱模态表示坍缩（MMC）的问题，提出几何感知框架GeoBalance，通过监测几何退化并触发重构，结合非对称梯度投影，在六个基准上显著提升均衡性能。
AI 中文摘要
多模态分类器可能收敛到模态主导的解，其中一种模态主导联合预测，抑制其他模态的学习。现有的均衡方法主要调整损失、梯度或模态贡献，大多将模态不平衡视为优化问题，同时隐含地认为弱模态是欠优化的但表示完整。在这项工作中，我们发现这一假设并不总是成立，因为持续的模态主导可诱发弱模态的表示级坍缩，我们称之为流形模态坍缩（MMC）。MMC表现为耦合的几何退化，其中弱模态表示在每个类内坍缩到较少的方向上，并且跨类的可分离性降低。受此观察启发，我们提出GeoBalance，一个几何感知框架，监测这两个几何属性，并仅在弱模态表示表现出MMC迹象时重构它。一旦触发，GeoBalance使用固定的Simplex-ETF类支架和谱正则化来恢复类间分离，同时防止坍缩到少数特征方向。为了在联合训练期间保持重构，非对称梯度投影移除与重构冲突的联合梯度分量，保留不冲突的优化不变。在六个多模态基准上的大量实验表明，相对于有竞争力的均衡方法有显著改进，验证了其有效性。
英文摘要
Multimodal classifiers can converge to modality-dominant solutions in which one modality dominates the joint prediction, suppressing the learning of others. Existing balancing methods mainly adjust losses, gradients, or modality contributions, largely treating modality imbalance as an optimization problem while implicitly treating the weak modality as under-optimized but representationally intact. In this work, we find that this assumption does not always hold, as persistent modality dominance can induce a representation-level collapse of the weak modality, which we term \emph{manifold modality collapse} (MMC). MMC manifests as a coupled geometric degradation in which weak-modality representations collapse onto fewer directions within each class and become less separable across classes. Motivated by this observation, we propose \emph{GeoBalance}, a geometry-aware framework that monitors these two geometric properties and reconstructs the weak modality representation only when it exhibits signs of MMC. Once triggered, GeoBalance uses a fixed Simplex-ETF class scaffold and spectral regularization to restore class separation while preventing collapse onto a few feature directions. To preserve reconstruction during joint training, asymmetric gradient projection removes the joint-gradient component conflicting with reconstruction, leaving non-conflicting optimization unchanged. Extensive experiments across six multimodal benchmarks demonstrate great improvements over competitive balancing methods, validating its effectiveness.
