arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.06799cs.CLcs.LG
收藏

让提示具备适应性:基于数据依赖先验的视觉-语言提示学习贝叶斯建模

Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior

KAIST(韩国科学技术院);Seoul National University(首尔大学);Ewha Womans University(梨花女子大学)

Youngjae Cho, HeeSun Bae, Seungjae Shin, Yeo Dong Youn, Weonyoung Joo, Il-Chul Moon

AI总结:

本文提出基于贝叶斯框架的视觉-语言提示学习方法,通过数据依赖先验和Wasserstein梯度流估计后验分布,缓解少样本场景下的过拟合,提升提示对未见实例的适应性,并在多个基准数据集上取得显著性能提升。

摘要:

近年来,视觉-语言预训练(VLP)模型已成为许多下游任务的骨干,但它们通常被当作冻结模型使用而不进行学习。提示学习是一种通过在文本编码器输入中添加可学习上下文向量来改进预训练VLP模型的方法。在下游任务的少样本学习场景中,MLE训练可能导致上下文向量过拟合训练数据中的主导图像特征。这种过拟合可能损害泛化能力,尤其是在训练集与测试集之间存在分布偏移的情况下。本文提出了一种基于贝叶斯框架的提示学习方法,可以缓解少样本学习应用中的过拟合问题,并提高提示对未见实例的适应性。具体而言,对数据依赖先验进行建模,能够在不在已见和未见图像特征之间进行性能权衡的情况下,增强文本特征对两者的适应性。基于该贝叶斯框架,我们在目标后验分布的估计中利用Wasserstein梯度流,使我们的提示能够灵活地捕捉图像特征的复杂模式。我们在基准数据集上通过若干实验证明了该方法的有效性,结果显示与现有方法相比,性能有统计学上的显著提升。代码可在 https://github.com/youngjae-cho/APP 获取。

英文摘要:

Recent Vision-Language Pretrained (VLP) models have become the backbone for many downstream tasks, but they are utilized as frozen model without learning. Prompt learning is a method to improve the pre-trained VLP model by adding a learnable context vector to the inputs of the text encoder. In a few-shot learning scenario of the downstream task, MLE training can lead the context vector to over-fit dominant image features in the training data. This overfitting can potentially harm the generalization ability, especially in the presence of a distribution shift between the training and test dataset. This paper presents a Bayesian-based framework of prompt learning, which could alleviate the overfitting issues on few-shot learning application and increase the adaptability of prompts on unseen instances. Specifically, modeling data-dependent prior enhances the adaptability of text features for both seen and unseen image features without the trade-off of performance between them. Based on the Bayesian framework, we utilize the Wasserstein Gradient Flow in the estimation of our target posterior distribution, which enables our prompt to be flexible in capturing the complex modes of image features. We demonstrate the effectiveness of our method on benchmark datasets for several experiments by showing statistically significant improvements on performance compared to existing methods. The code is available at https://github.com/youngjae-cho/APP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑