基于跨模态适配器与知识保留的表示对齐的多模态持续学习
Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation
AI总结：
针对多模态持续学习需整合多模态新信息且防止灾难性遗忘的问题，提出含专家混合结构跨模态适配器、表示对齐损失及知识保留正则化的预训练模型框架，在类/域增量学习中准确率更高且遗忘更少。
摘要：
持续学习对于模型在保留已有知识的同时适应新任务至关重要。现有方法主要聚焦于单模态数据，而多模态学习通过利用多样化的感官输入（类似人类感知）提供了显著优势。然而，多模态持续学习面临额外挑战：模型必须有效整合来自不同模态的新信息，同时防止灾难性遗忘。在这项工作中，我们提出了一个基于预训练模型的多模态持续学习框架。该框架包含一个具有专家混合结构的新型跨模态适配器，以促进跨任务多模态信息的有效整合；还引入了表示对齐损失，用于学习鲁棒的多模态表示；并通过正则化已学习表示之间的关系来保留先前任务的知识。在多个多模态数据集上的实验表明，我们的方法在类增量和域增量学习中持续优于基线方法，实现了更高的准确率并减少了遗忘。
英文摘要：
Continual learning is essential for adapting models to new tasks while retaining previously acquired knowledge. While existing approaches predominantly focus on uni-modal data, multi-modal learning offers substantial benefits by utilizing diverse sensory inputs, akin to human perception. However, multi-modal continual learning presents additional challenges, as the model must effectively integrate new information from various modalities while preventing catastrophic forgetting. In this work, we propose a pre-trained model-based framework for multi-modal continual learning. Our framework includes a novel cross-modality adapter with a mixture-of-experts structure to facilitate effective integration of multi-modal information across tasks. We also introduce a representation alignment loss that fosters learning of robust multi-modal representations, and regularize relationships between learned representations to preserve knowledge from previous tasks. Experiments on several multi-modal datasets demonstrate that our approach consistently outperforms baselines in both class-incremental and domain-incremental learning, achieving higher accuracy and reduced forgetting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
