持续多模态对比学习
Continual Multimodal Contrastive Learning
AI总结：
本文提出持续多模态对比学习（CMCL）问题，基于稳定性与可塑性原则推导出将双侧梯度投影到无干扰子空间的优化方法，并通过理论分析和多数据集实验验证了其有效性。
摘要：
多模态对比学习（MCL）在对齐不同模态并在联合空间中生成多模态表示方面取得了进展。通过利用跨不同模态的对比学习，大规模多模态数据提升了表示质量。然而，一个关键但常被忽视的挑战仍然存在：多模态数据很少在单一过程中收集，从头训练的计算成本很高。相反，新出现的多模态数据可以用于逐步优化现有模型，即模型在一系列模态对数据上进行训练。我们将这一问题定义为持续多模态对比学习（CMCL），这是一个在多模态学习与持续学习交叉领域中尚未被充分探索但至关重要的研究方向。在本文中，我们通过稳定性和可塑性这两个专门原则来形式化 CMCL。我们从理论上推导出一种新颖的基于优化的方法，该方法将来自双侧的更新梯度投影到子空间上，在这些子空间中任何梯度都不会干扰先前学到的知识。两个上界为我们的解决方案在稳定性和可塑性方面提供了理论见解。除了理论贡献之外，我们还在多个数据集上进行了实验，将我们的方法与先进的持续学习基线进行了比较。实证结果进一步支持了我们的主张，并证明了我们方法的有效性。我们的代码可在 https://github.com/Xiaohao-Liu/CMCL 获取。
英文摘要：
Multimodal Contrastive Learning (MCL) advances in aligning different modalities and generating multimodal representations in a joint space. By leveraging contrastive learning across diverse modalities, large-scale multimodal data enhances representational quality. However, a critical yet often overlooked challenge remains: multimodal data is rarely collected in a single process, and training from scratch is computationally expensive. Instead, emergent multimodal data can be used to optimize existing models gradually, i.e., models are trained on a sequence of modality pair data. We define this problem as Continual Multimodal Contrastive Learning (CMCL), an underexplored yet crucial research direction at the intersection of multimodal and continual learning. In this paper, we formulate CMCL through two specialized principles of stability and plasticity. We theoretically derive a novel optimization-based method, which projects updated gradients from dual sides onto subspaces where any gradient is prevented from interfering with the previously learned knowledge. Two upper bounds provide theoretical insights on both stability and plasticity in our solution. Beyond our theoretical contributions, we conduct experiments on multiple datasets by comparing our method against advanced continual learning baselines. The empirical results further support our claims and demonstrate the efficacy of our method. Our codes are available at https://github.com/Xiaohao-Liu/CMCL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
