arXiv 2503.186722025-10-28cs.CV

CalFuse：基于特征校准与参数融合的多模态持续学习

CalFuse: Multi-Modal Continual Learning via Feature Calibration and Parameter Fusion

College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；School of Future Science and Engineering, Soochow University（未来科学与工程学院，苏州大学）；DataLab: Data Science and Informatics, University of California, Davis（数据实验室：数据科学与信息学，加州大学戴维斯分校）；University of Rochester（罗切斯特大学）；Ningbo University（宁波大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing University of Posts and Telecommunications（北京邮电大学）；Academy for Computer Science and Informatics, Cardiff University（计算机科学与信息学学院，卡迪夫大学）

Juncen Guo, Siao Liu, Xiaoguang Zhu, Lianlong Sun, Liangyu Teng, Jingyi Wu, Di Li, Linxiao Gong, Weiwei Jiang, Wei Zhou, Liang Song

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态持续学习中灾难性遗忘与跨模态泛化难以兼顾的问题，提出融合特征校准与QR分解参数融合的CalFuse框架，在基准数据集上的平均准确率和任务保留率均优于现有方法。

摘要：

随着大规模视觉识别系统中多模态数据的激增，使模型能够在不断演变的数据流中持续获取知识同时保留已有信息变得愈发关键。类别持续学习（Class-Continual Learning, CCL）通过增量式融入新类别知识且无需重新访问历史数据来应对这一挑战，这对现实世界的大数据应用至关重要。传统CCL方法仅依赖视觉特征，而CLIP等视觉语言模型（Vision-Language Models, VLMs）的最新进展表明，利用预训练的多模态知识在CCL中具有巨大潜力。然而，现有方法在缓解灾难性遗忘的同时维持VLM的跨模态泛化能力方面仍面临挑战。为解决这些局限，我们提出CalFuse框架，该框架将特征校准与参数融合协同结合，以在持续学习场景中实现有效的多模态知识整合。CalFuse引入了动态特征校准机制，可自适应平衡原始CLIP视觉表示与任务特定特征，在适应新类别的同时保留模型固有的跨模态泛化能力。同时，基于QR分解的参数融合策略将新获取的知识与历史任务参数逐步整合，在连续任务中维持学习新类别表示与保留先验知识之间的平衡。在基准数据集上开展的大量实验验证了我们的方法在大规模多模态持续学习设置中的有效性，表明其在平均准确率和最终任务保留率上均优于当前最优方法。

英文摘要：

With the proliferation of multi-modal data in large-scale visual recognition systems, enabling models to continuously acquire knowledge from evolving data streams while preserving prior information has become increasingly critical. Class-Continual Learning (CCL) addresses this challenge by incrementally incorporating new class knowledge without revisiting historical data, making it essential for real-world big data applications. While traditional CCL methods rely solely on visual features, recent advances in Vision-Language Models (VLMs) such as CLIP demonstrate significant potential for CCL by leveraging pre-trained multi-modal knowledge. However, existing approaches face challenges in mitigating catastrophic forgetting while maintaining the cross-modal generalization capabilities of VLMs. To address these limitations, we propose CalFuse, a framework that synergizes feature Calibration with parameter Fusion to enable effective multi-modal knowledge integration in continual learning scenarios. CalFuse introduces a dynamic feature calibration mechanism that adaptively balances original CLIP visual representations with task-specific features, preserving the model's intrinsic cross-modal generalization while adapting to new classes. Concurrently, a QR decomposition-based parameter fusion strategy progressively integrates newly acquired knowledge with historical task parameters, maintaining equilibrium between learning new class representations and retaining prior knowledge across sequential tasks. Extensive experiments on benchmark datasets validate the effectiveness of our approach in large-scale multi-modal continual learning settings, demonstrating superior performance over state-of-the-art methods in both average accuracy and final task retention.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。