arXiv 2502.143762025-06-03cs.CLcs.CV

一种通过文本正则化实现不遗忘的相似性范式

A Similarity Paradigm Through Textual Regularization Without Forgetting

Shanghai Jiao Tong University（上海交通大学）；Hong Kong Baptist University（香港浸会大学）；Northeastern University（东北大学）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））

Fangming Cui, Jan Fong, Rongfei Zeng, Xinmei Tian, Jun Yu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出带文本正则化的相似性范式SPTR，通过最优传输正则化和自然/对抗对齐分数，缓解提示学习中的遗忘问题并提升泛化能力，在11个数据集上优于现有方法。

摘要：

提示学习已成为一种有前景的方法，用于将预训练的视觉-语言模型（VLMs）适配到一系列下游任务。虽然优化上下文对于提升特定任务上的性能可能有效，但它往往会导致在未见类别或来自不同分布的数据集上的泛化性能较差。这可能归因于文本提示倾向于过拟合下游数据分布，从而导致遗忘从手工设计提示中获得的泛化知识。在本文中，我们提出了一种名为“带文本正则化的相似性范式”（SPTR）的新方法，用于实现不遗忘的提示学习。SPTR 是一种基于手工设计提示的双管齐下设计，构成一个不可分割的框架。1）为避免遗忘通用文本知识，我们引入最优传输作为文本正则化，以精细地确保手工设计特征与调优文本特征之间的近似。2）为了持续释放多个手工设计提示的通用能力，我们提出了一种用于自然对齐分数和对抗对齐分数的相似性范式，以提升模型在泛化方面的鲁棒性。两个模块在解决泛化问题上具有共同目标，旨在最大化从多个手工设计提示中获得的泛化能力。在 11 个数据集上的四项代表性任务（即非泛化少样本学习、基类到新类泛化、跨数据集泛化、域泛化）表明，SPTR 优于现有的提示学习方法。

英文摘要：

Prompt learning has emerged as a promising method for adapting pre-trained visual-language models (VLMs) to a range of downstream tasks. While optimizing the context can be effective for improving performance on specific tasks, it can often lead to poor generalization performance on unseen classes or datasets sampled from different distributions. It may be attributed to the fact that textual prompts tend to overfit downstream data distributions, leading to the forgetting of generalized knowledge derived from hand-crafted prompts. In this paper, we propose a novel method called Similarity Paradigm with Textual Regularization (SPTR) for prompt learning without forgetting. SPTR is a two-pronged design based on hand-crafted prompts that is an inseparable framework. 1) To avoid forgetting general textual knowledge, we introduce the optimal transport as a textual regularization to finely ensure approximation with hand-crafted features and tuning textual features. 2) In order to continuously unleash the general ability of multiple hand-crafted prompts, we propose a similarity paradigm for natural alignment score and adversarial alignment score to improve model robustness for generalization. Both modules share a common objective in addressing generalization issues, aiming to maximize the generalization capability derived from multiple hand-crafted prompts. Four representative tasks (i.e., non-generalization few-shot learning, base-to-novel generalization, cross-dataset generalization, domain generalization) across 11 datasets demonstrate that SPTR outperforms existing prompt learning methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。