arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.04489cs.CV
收藏

Dude:面向大型视觉-语言模型的双重分布感知上下文提示学习

Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model

University of Stuttgart(斯图加特大学);German Research Center for Artificial Intelligence(德国人工智能研究中心);Max Planck Research School for Intelligent Systems(马克斯·普朗克智能系统研究学校);Technical University of Darmstadt(达姆施塔特工业大学);Technical University of Munich(慕尼黑工业大学);United States Naval Academy(美国海军学院);Oldenburg University(奥尔登堡大学);University of Pennsylvania(宾夕法尼亚大学)

Duy M. H. Nguyen, An T. Le, Trung Q. Nguyen, Nghiem T. Diep, Tai Nguyen, Duy Duong-Tran, Jan Peters, Li Shen, Mathias Niepert, Daniel Sonntag

AI总结:

本文提出Dude框架,通过域共享与类特定双重上下文提示学习,并引入非平衡最优传输理论对齐视觉令牌与提示嵌入,提升大型视觉-语言模型在细粒度少样本分类任务中的性能。

摘要:

提示学习方法因其能够利用预训练的上下文知识和极少量的训练数据将大型视觉-语言模型定制到新领域而受到越来越多的关注。然而,现有工作通常依赖于优化统一的提示输入,由于判别性属性不足,往往在细粒度分类任务上表现不佳。为了解决这一问题,我们考虑了一个基于域共享上下文和类特定上下文双重语境的新框架,其中类特定上下文由大型语言模型(LLMs)如GPT生成。这种双重提示方法通过结合编码在LLM知识中的隐式和显式因素,增强了模型的特征表示能力。此外,我们构建了非平衡最优传输(UOT)理论来量化所构建提示与视觉令牌之间的关系。通过部分匹配,UOT能够在不同质量分布下恰当地对齐离散的视觉令牌集合与提示嵌入,这对于处理不相关或噪声元素尤为有价值,确保质量守恒不会限制传输解。此外,UOT的特性与图像增强无缝集成,在扩大训练样本池的同时保持扰动图像与提示输入之间的合理距离。在少样本分类和适配器设置下的大量实验证实了我们的模型相对于当前最先进基线的优越性。

英文摘要:

Prompt learning methods are gaining increasing attention due to their ability to customize large vision-language models to new domains using pre-trained contextual knowledge and minimal training data. However, existing works typically rely on optimizing unified prompt inputs, often struggling with fine-grained classification tasks due to insufficient discriminative attributes. To tackle this, we consider a new framework based on a dual context of both domain-shared and class-specific contexts, where the latter is generated by Large Language Models (LLMs) such as GPTs. Such dual prompt methods enhance the model's feature representation by joining implicit and explicit factors encoded in LLM knowledge. Moreover, we formulate the Unbalanced Optimal Transport (UOT) theory to quantify the relationships between constructed prompts and visual tokens. Through partial matching, UOT can properly align discrete sets of visual tokens and prompt embeddings under different mass distributions, which is particularly valuable for handling irrelevant or noisy elements, ensuring that the preservation of mass does not restrict transport solutions. Furthermore, UOT's characteristics integrate seamlessly with image augmentation, expanding the training sample pool while maintaining a reasonable distance between perturbed images and prompt inputs. Extensive experiments across few-shot classification and adapter settings substantiate the superiority of our model over current state-of-the-art baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑