arXiv 2505.180532025-05-23cs.CVcs.AI

FDBPL：用于区域感知视觉-语言模型适配的更快基于蒸馏的提示学习

FDBPL: Faster Distillation-Based Prompt Learning for Region-Aware Vision-Language Models Adaptation

School of Artificial Intelligence, Beijing University of Posts；School of National Elite Institute of Engineering, Chongqing University, Chongqing, China；The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China；The Key Laboratory of Computing Power Network；Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology, Jinan, China.；Shandong Provincial Key Laboratory of Computing Power Internet；Service Computing, Shandong Fundamental Research Center for Computer Science, Jinan, China. -1cm

Zherui Zhang, Jiaxin Wu, Changwei Wang, Rongtao Xu, Longzhao Huang, Wenhao Xu, Wenbo Xu, Li Guo, Shibiao Xu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出FDBPL，一种更快的基于蒸馏的提示学习方法，通过共享软监督上下文和加速I/O，结合区域感知的双正负提示空间，在保持参数效率的同时提升视觉-语言模型适配的零样本泛化性能，训练速度提升2.2倍。

摘要：

提示学习作为一种参数高效的方法，已被广泛用于将视觉-语言模型（VLMs）适配到下游任务。虽然硬提示设计需要领域专业知识和迭代优化，但软提示方法严重依赖任务特定的硬标签，限制了其对未见类别的泛化能力。近期流行的基于蒸馏的提示学习方法通过利用更大的教师VLM和无监督知识转移来提高泛化能力，但其重复的教师模型在线推理牺牲了提示学习固有的训练效率优势。在本文中，我们提出了更快的基于蒸馏的提示学习（FDBPL），通过跨多个训练阶段共享软监督上下文并实现加速的输入/输出（I/O）来解决这些问题。此外，FDBPL引入了一种区域感知的提示学习范式，具有双正负提示空间，以充分利用包含多级信息的随机裁剪区域。我们提出了一种基于相似性-差异性学习的正负空间相互学习机制，使学生CLIP模型能够识别正确的语义，同时学习拒绝弱相关概念，从而提高零样本性能。与现有的以牺牲参数效率换取泛化能力的基于蒸馏的提示学习方法不同，FDBPL保持了参数效率和强下游泛化的双重优势。在11个数据集上的全面评估表明，在基础到新类泛化、跨数据集迁移和鲁棒性测试中均表现出优越性能，训练速度提高了2.2倍。

英文摘要：

Prompt learning as a parameter-efficient method that has been widely adopted to adapt Vision-Language Models (VLMs) to downstream tasks. While hard-prompt design requires domain expertise and iterative optimization, soft-prompt methods rely heavily on task-specific hard labels, limiting their generalization to unseen categories. Recent popular distillation-based prompt learning methods improve generalization by exploiting larger teacher VLMs and unsupervised knowledge transfer, yet their repetitive teacher model online inference sacrifices the inherent training efficiency advantage of prompt learning. In this paper, we propose {\large {\textbf{F}}}aster {\large {\textbf{D}}}istillation-{\large {\textbf{B}}}ased {\large {\textbf{P}}}rompt {\large {\textbf{L}}}earning (\textbf{FDBPL}), which addresses these issues by sharing soft supervision contexts across multiple training stages and implementing accelerated I/O. Furthermore, FDBPL introduces a region-aware prompt learning paradigm with dual positive-negative prompt spaces to fully exploit randomly cropped regions that containing multi-level information. We propose a positive-negative space mutual learning mechanism based on similarity-difference learning, enabling student CLIP models to recognize correct semantics while learning to reject weakly related concepts, thereby improving zero-shot performance. Unlike existing distillation-based prompt learning methods that sacrifice parameter efficiency for generalization, FDBPL maintains dual advantages of parameter efficiency and strong downstream generalization. Comprehensive evaluations across 11 datasets demonstrate superior performance in base-to-new generalization, cross-dataset transfer, and robustness tests, achieving $2.2\times$ faster training speed.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。