CLAP4CLIP: 面向视觉-语言模型的基于概率微调的持续学习
CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models
AI总结:
针对预训练视觉语言模型CLIP在持续学习微调中确定性方法的不足,提出概率微调框架CLAP,利用预训练知识缓解遗忘,性能超越主流确定性方法,还具备优秀的不确定性估计能力。
摘要:
持续学习(CL)旨在帮助深度神经网络在学习新知识的同时保留已学内容。凭借强大的泛化能力,Contrastive Language-Image Pre-training(CLIP,对比语言-图像预训练)等预训练视觉-语言模型近期成为实用的持续学习候选方案,受到广泛关注。然而,预训练与下游持续学习任务之间存在域偏移,通常需要在后者上对CLIP进行微调。现有大多数微调方法具有确定性本质,这使其忽略了输入模态间多种可能的交互作用,且在需要可靠不确定性估计的高风险任务中存在安全隐患。为解决这些问题,本研究提出了Continual LeArning with Probabilistic finetuning(CLAP,基于概率微调的持续学习)——这是一个针对每个任务的视觉引导文本特征的概率建模框架,可提供更校准的持续学习微调效果。与近期依赖大量数据的抗遗忘持续学习技术不同,CLAP利用CLIP丰富的预训练知识进行权重初始化,并对任务特定参数进行分布正则化,从而缓解遗忘问题。CLAP可与现有各类提示方法协同使用,在基于CLIP的持续学习任务中性能超越主流的确定性微调方法。最后,我们展示了CLAP卓越的不确定性估计能力的开箱即用应用,包括在现有持续学习设置下的新数据检测和样本选择。我们的代码可在https://github.com/srvCodes/clap4clip 获取。
英文摘要:
Continual learning (CL) aims to help deep neural networks learn new knowledge while retaining what has been learned. Owing to their powerful generalizability, pre-trained vision-language models such as Contrastive Language-Image Pre-training (CLIP) have lately gained traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks often calls for finetuning of the CLIP on the latter. Most existing finetuning methods exhibit deterministic nature. This makes them overlook the many possible interactions across the input modalities and deems them unsafe for high-risk tasks requiring reliable uncertainty estimation. To address these, our work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task, thus providing more calibrated CL finetuning. Unlike recent data-hungry anti-forgetting CL techniques, CLAP alleviates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific parameters. Cooperating with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-box applications of superior uncertainty estimation abilities of CLAP including novel data detection and exemplar selection within the existing CL setups. Our code is available at \url{https://github.com/srvCodes/clap4clip}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。