利用非对角信息的持续视觉-语言表征学习
Continual Vision-Language Representation Learning with Off-Diagonal Information
AI总结：
本文研究流式数据下持续CLIP训练的性能退化问题，将其归因于空间紊乱，并提出Mod-X通过选择性维护对比矩阵的非对角信息来保持旧域多模态对齐。
摘要：
像CLIP这样的大规模多模态对比学习框架通常需要大量图文样本进行训练。然而，在真实场景中，这些样本总是持续收集的。本文讨论了使用流式数据进行持续CLIP训练的可行性。不同于针对纯图像、基于自监督学习方法的持续学习（据经验观察，其对灾难性遗忘具有鲁棒性），CLIP在持续设定下的性能退化十分显著且不可忽视。通过从空间几何视角分析持续CLIP训练期间模型表征空间的变化，我们将这些空间变化探索并总结为空间紊乱（Spatial Disorder，SD），其可划分为模态内旋转和模态间偏移。此外，我们从经验和理论上证明了SD如何导致CLIP在跨模态检索任务上的性能下降。为缓解SD，我们提出了一种新的持续视觉-语言表征学习框架Mod-X：维护非对角信息矩阵（Maintain off-diagonal information-matriX）。通过选择性地对齐对比矩阵的非对角信息分布，Mod-X在持续拟合新训练数据域期间，维护旧数据域上的多模态表征空间对齐，从而提升多模态模型的能力。在具有不同规模和范围的常用数据集上进行的实验证明了我们方法的有效性。
英文摘要：
Large-scale multi-modal contrastive learning frameworks like CLIP typically require a large amount of image-text samples for training. However, these samples are always collected continuously in real scenarios. This paper discusses the feasibility of continual CLIP training using streaming data. Unlike continual learning based on self-supervised learning methods for pure images, which is empirically robust against catastrophic forgetting, CLIP's performance degeneration in the continual setting is significant and non-neglectable. By analyzing the changes in the model's representation space during continual CLIP training from a spatial geometry perspective, we explore and summarize these spatial variations as Spatial Disorder (SD), which can be divided into Intra-modal Rotation and Inter-modal Deviation. Moreover, we empirically and theoretically demonstrate how SD leads to a performance decline for CLIP on cross-modal retrieval tasks. To alleviate SD, we propose a new continual vision-language representation learning framework Mod-X: Maintain off-diagonal information-matriX. By selectively aligning the off-diagonal information distribution of contrastive matrices, the Mod-X improves the capability of the multi-modal model by maintaining the multi-modal representation space alignment on the old data domain during continuously fitting the new training data domain. Experiments on commonly used datasets with different scales and scopes have demonstrated the effectiveness of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
