arXiv 2403.102452024-03-15cs.CV

CoLeCLIP：通过联合任务提示与词汇学习的开放域持续学习

CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learning

Northwestern Polytechnical University（西北工业大学）；Ningbo Institute（宁波研究院）；Singapore Management University（新加坡管理大学）；Zhejiang University（浙江大学）；University of Adelaide（阿德莱德大学）

Yukun Li, Guansong Pang, Wei Suo, Chenchen Jing, Yuling Xi, Lingqiao Liu, Hao Chen, Guoqiang Liang, Peng Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CoLeCLIP方法，通过联合学习任务提示和跨领域类别词汇，解决开放域中视觉-语言模型的持续学习问题，在11个数据集上超越现有方法。

摘要：

本文探讨了开放域中视觉-语言模型（VLMs）的持续学习（CL）问题，在该问题中，模型需要对来自多样化的已知和未知领域的流式数据集进行持续更新和推理，且这些数据集包含新类别。这种能力对于开放环境中的各种应用至关重要，例如AI助手、自动驾驶系统和机器人技术。当前的持续学习研究大多集中于单一领域中已知类别的封闭集场景。像CLIP这样的大型预训练视觉-语言模型已展现出卓越的零样本识别能力，近期许多研究利用这一能力来缓解持续学习中的灾难性遗忘，但它们主要关注单一领域数据集上的封闭集持续学习。大型视觉-语言模型的开放域持续学习更具挑战性，原因在于：1）各数据集之间存在较大的类别相关性和领域差距；2）除了从新适应的数据集中学到的知识外，预训练视觉-语言模型中的零样本知识也会被遗忘。在这项工作中，我们提出了一种名为CoLeCLIP的新方法，基于CLIP学习开放域持续学习模型。它通过联合学习一组任务提示和跨领域类别词汇表来应对这些挑战。在11个领域数据集上的大量实验表明，CoLeCLIP在任务增量学习和类别增量学习两种设置下均优于开放域持续学习的最先进方法。

英文摘要：

This paper explores the problem of continual learning (CL) of vision-language models (VLMs) in open domains, where the models need to perform continual updating and inference on a streaming of datasets from diverse seen and unseen domains with novel classes. Such a capability is crucial for various applications in open environments, e.g., AI assistants, autonomous driving systems, and robotics. Current CL studies mostly focus on closed-set scenarios in a single domain with known classes. Large pre-trained VLMs like CLIP have demonstrated superior zero-shot recognition ability, and a number of recent studies leverage this ability to mitigate catastrophic forgetting in CL, but they focus on closed-set CL in a single domain dataset. Open-domain CL of large VLMs is significantly more challenging due to 1) large class correlations and domain gaps across the datasets and 2) the forgetting of zero-shot knowledge in the pre-trained VLMs in addition to the knowledge learned from the newly adapted datasets. In this work we introduce a novel approach, termed CoLeCLIP, that learns an open-domain CL model based on CLIP. It addresses these challenges by a joint learning of a set of task prompts and a cross-domain class vocabulary. Extensive experiments on 11 domain datasets show that CoLeCLIP outperforms state-of-the-art methods for open-domain CL under both task- and class-incremental learning settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。