arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.06136cs.CV
收藏

RESTORE: 面向视觉语言提示学习的特征偏移研究

RESTORE: Towards Feature Shift for Vision-Language Prompt Learning

Shanghai Jiao Tong University(上海交通大学);Guangzhou University(广州大学);Tencent Youtu Lab(腾讯优图实验室)

Yuncheng Yang, Chuyan Zhang, Zuopeng Yang, Yuting Gao, Yulei Qin, Ke Li, Xing Sun, Jie Yang, Yun Gu

AI总结:

针对单模态提示调优导致CLIP视觉-语言对齐受损、泛化下降的问题,提出RESTORE多模态提示学习方法,通过特征偏移一致性约束和“手术”适配器模块缓解不对齐,在15个数据集上优于现有方法且保持特征对齐。

摘要:

提示学习对基础模型的微调十分有效,能够提升其在各类下游任务上的泛化能力。然而,沿单模态路径独立优化的提示,可能会牺牲预训练模型的视觉-语言对齐性,以换取特定任务和类别上的性能提升,最终导致泛化能力下降。本文首先证明,仅沿CLIP(对比语言-图像预训练)的单个分支(例如语言或视觉分支)进行提示调优是出现这种不对齐问题的原因。由于不同模态的可学习参数之间缺乏适当的正则化,提示学习违反了双塔架构固有的原始预训练约束。为解决这种不对齐问题,我们首先提出特征偏移的概念,定义为引入学习到的提示后嵌入的变化量,将其作为一种解释工具。我们深入探究了它与泛化能力的关系,随后提出了RESTORE——一种对跨模态一致性施加显式约束的多模态提示学习方法。具体而言,为防止特征不对齐,我们引入特征偏移一致性机制,通过在提示调优过程中测量并正则化差异的幅度,来同步模态间的特征偏移。此外,我们提出了一个“手术”模块,以避免捷径问题——当各模态的特征偏移以相同速率剧烈变化时,跨模态不对齐问题仍然可能十分严重。该模块以两个模态上的前馈适配器形式实现,用于缓解不对齐问题。在15个数据集上开展的大量实验表明,我们的方法在不损害特征对齐性的前提下,性能优于现有最先进的提示调优方法。

英文摘要:

Prompt learning is effective for fine-tuning foundation models to improve their generalization across a variety of downstream tasks. However, the prompts that are independently optimized along a single modality path, may sacrifice the vision-language alignment of pre-trained models in return for improved performance on specific tasks and classes, leading to poorer generalization. In this paper, we first demonstrate that prompt tuning along only one single branch of CLIP (e.g., language or vision) is the reason why the misalignment occurs. Without proper regularization across the learnable parameters in different modalities, prompt learning violates the original pre-training constraints inherent in the two-tower architecture. To address such misalignment, we first propose feature shift, which is defined as the variation of embeddings after introducing the learned prompts, to serve as an explanatory tool. We dive into its relation with generalizability and thereafter propose RESTORE, a multi-modal prompt learning method that exerts explicit constraints on cross-modal consistency. To be more specific, to prevent feature misalignment, a feature shift consistency is introduced to synchronize inter-modal feature shifts by measuring and regularizing the magnitude of discrepancy during prompt tuning. In addition, we propose a "surgery" block to avoid short-cut hacking, where cross-modal misalignment can still be severe if the feature shift of each modality varies drastically at the same rate. It is implemented as feed-forward adapters upon both modalities to alleviate the misalignment problem. Extensive experiments on 15 datasets demonstrate that our method outperforms the state-of-the-art prompt tuning methods without compromising feature alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑