DesCLIP：通过通用属性描述实现基于视觉语言模型的鲁棒持续学习
DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognition
AI总结：
本文提出DesCLIP，通过通用属性描述引导视觉语言模型理解特定类别对象，建立视觉-通用属性-类别三元关联，提升持续学习效果。
摘要：
视觉语言模型（VLM）的持续学习旨在利用跨模态预训练知识逐步适应扩展的下游任务和数据集，同时解决知识遗忘问题。现有研究常关注将视觉特征与特定类别文本连接，忽视了通用知识与专业知识之间的潜在关系。我们的发现表明，强制模型优化不恰当的视觉-文本匹配会加剧VLM识别能力的遗忘。为此，我们提出DesCLIP，利用通用属性（GA）描述指导对特定类别对象的理解，使VLM建立稳健的视觉-GA-类别三元关联，而非仅依赖视觉-类别连接。具体而言，我们引入语言助手生成具体的GA描述候选，然后设计基于锚点的嵌入过滤器获取高度相关的GA描述嵌入，这些嵌入被用作视觉-文本实例匹配的配对文本嵌入，从而调整视觉编码器。相应地，类别文本嵌入逐渐校准以与这些共享的GA描述嵌入对齐。广泛实验验证了所提方法的进展和有效性，全面的实证评估突显其在基于VLM的识别中优于现有持续学习方法的优越性能。
英文摘要：
Continual learning of vision-language models (VLMs) focuses on leveraging cross-modal pretrained knowledge to incrementally adapt to expanding downstream tasks and datasets, while tackling the challenge of knowledge forgetting. Existing research often focuses on connecting visual features with specific class text in downstream tasks, overlooking the latent relationships between general and specialized knowledge. Our findings reveal that forcing models to optimize inappropriate visual-text matches exacerbates forgetting of VLM's recognition ability. To tackle this issue, we propose DesCLIP, which leverages general attribute (GA) descriptions to guide the understanding of specific class objects, enabling VLMs to establish robust vision-GA-class trilateral associations rather than relying solely on vision-class connections. Specifically, we introduce a language assistant to generate concrete GA description candidates via proper request prompts. Then, an anchor-based embedding filter is designed to obtain highly relevant GA description embeddings, which are leveraged as the paired text embeddings for visual-textual instance matching, thereby tuning the visual encoder. Correspondingly, the class text embeddings are gradually calibrated to align with these shared GA description embeddings. Extensive experiments demonstrate the advancements and efficacy of our proposed method, with comprehensive empirical evaluations highlighting its superior performance in VLM-based recognition compared to existing continual learning methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
