arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.08708cs.CV
收藏

FVG-PT:适应性前景视图引导的提示微调用于视觉-语言模型

FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

University of Technology Sydney(悉尼技术大学);Shanghai University(上海大学)

Haoyang Li, Liang Wang, Siyu Zhou, Jiacheng Sun, Jing Jiang, Chao Wang, Guodong Long, Yan Peng

AI总结:

FVG-PT通过引入前景可靠性门、蒸馏补偿模块和先验校准模块,解决提示微调中前景注意力偏移问题,提升视觉-语言模型的适应性和泛化能力。

摘要:

基于CLIP的提示微调使预训练的视觉-语言模型(VLMs)能够高效适应下游任务。尽管现有研究已取得显著进展,但它们对微调过程中VLMs内部注意力表示的变化关注有限。本文将提示微调预测失败模式归因于视觉编码器中前景注意力的偏移,并提出前景视图引导的提示微调(FVG-PT),一种适应性强的插件式前景注意力指导模块,以缓解此类偏移。具体而言,FVG-PT引入了一个可学习的前景可靠性门,自动提升前景视图质量;应用前景蒸馏补偿模块,引导视觉注意力朝向前景;并进一步引入先验校准模块,以减轻因过度关注前景导致的泛化退化。在多个主干模型和数据集上的实验表明了FVG-PT的有效性和兼容性。代码可在:https://github.com/JREion/FVG-PT获取。

英文摘要:

CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑