arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2210.01115cs.CVcs.AIcs.LG
收藏

LASP:面向视觉与语言模型的语言感知软提示的文本到文本优化

LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models

Samsung AI Cambridge(三星剑桥人工智能研究院);Technical University of Iasi(雅西技术大学);Queen Mary University of London(伦敦玛丽女王大学)

Adrian Bulat, Georgios Tzimiropoulos

AI总结:

本文提出LASP方法通过文本到文本交叉熵损失缓解视觉语言模型的软提示过拟合,并引入分组优化、重新校准机制和虚拟类增强鲁棒性,在11个数据集上显著超越现有工作并首次在8个数据集上超越手工提示和CLIP的未见类准确率。

摘要:

软提示学习近期已成为使用少量训练样本将V&L模型适配到下游任务的首选方法之一。然而,当前方法严重过拟合训练数据,在对来自同一域的未见类别进行测试时,准确率大幅下降。为此,本文做出以下4项贡献:(1)为缓解基类过拟合,我们提出一种新颖的语言感知软提示(LASP)学习方法,通过文本到文本交叉熵损失,最大化学习到的提示相对于预定义的手工文本提示被正确分类的概率。(2)为增加提示的表示能力,我们提出分组LASP,其中每组提示针对单独的文本提示子集进行优化。(3)我们识别出由提示学习和LASP引入的视觉-语言错位,并更重要的是,提出一种重新校准机制来解决该问题。(4)我们表明LASP在本质上支持在训练期间引入虚拟类(即没有视觉样本可用的类别名称),从而进一步提高学习到的提示的鲁棒性。通过对11个数据集的评估,我们表明我们的方法(a)显著优于所有先前的软提示工作,并且(b)在11个测试数据集中有8个首次匹配并超越了由手工提示和CLIP获得的未见类准确率。代码将在https://www.adrianbulat.com/lasp上发布。

英文摘要:

Soft prompt learning has recently emerged as one of the methods of choice for adapting V&L models to a downstream task using a few training examples. However, current methods significantly overfit the training data, suffering from large accuracy degradation when tested on unseen classes from the same domain. To this end, in this paper, we make the following 4 contributions: (1) To alleviate base class overfitting, we propose a novel Language-Aware Soft Prompting (LASP) learning method by means of a text-to-text cross-entropy loss that maximizes the probability of the learned prompts to be correctly classified with respect to pre-defined hand-crafted textual prompts. (2) To increase the representation capacity of the prompts, we propose grouped LASP where each group of prompts is optimized with respect to a separate subset of textual prompts. (3) We identify a visual-language misalignment introduced by prompt learning and LASP, and more importantly, propose a re-calibration mechanism to address it. (4) We show that LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available, further increasing the robustness of the learned prompts. Through evaluations on 11 datasets, we show that our approach (a) significantly outperforms all prior works on soft prompting, and (b) matches and surpasses, for the first time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for 8 out of 11 test datasets. Code will be made available at https://www.adrianbulat.com/lasp

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑