通过专用提示引导泛化视觉语言模型
Generalizing Vision-Language Models with Dedicated Prompt Guidance
AI总结:
本文提出GuiDG框架,通过提示微调和跨模态注意力模块提升视觉语言模型的领域泛化能力,并在ImageNet-DG上验证了其有效性。
摘要:
对大规模预训练视觉语言模型(VLMs)进行微调已成为下游适应的主流范式,但面临领域特异性与领域泛化(DG)能力之间的关键权衡。当前方法通常在完整数据集上微调通用模型,这可能会削弱对未见领域的泛化能力。为填补这一空白,我们提供了VLM微调泛化能力的理论理解,揭示了在分域源数据上训练多个参数高效专家模型比微调通用模型更能提高泛化能力。受此发现启发,我们提出了一种两阶段领域专家引导的DG(GuiDG)框架。GuiDG首先使用提示微调获得源领域专家,然后引入跨模态注意力模块通过自适应专家整合来引导视觉编码器的微调。为更好地评估少样本DG,我们从ImageNet构建ImageNet-DG及其变体。在标准DG基准和ImageNet-DG上的广泛实验表明,GuiDG在保持效率的同时优于现有微调方法。
英文摘要:
Fine-tuning large pretrained vision-language models (VLMs) has emerged as a prevalent paradigm for downstream adaptation, yet it faces a critical trade-off between domain specificity and domain generalization (DG) ability. Current methods typically fine-tune a universal model on the entire dataset, which potentially compromises the ability to generalize to unseen domains. To fill this gap, we provide a theoretical understanding of the generalization ability for VLM fine-tuning, which reveals that training multiple parameter-efficient expert models on partitioned source domains leads to better generalization than fine-tuning a universal model. Inspired by this finding, we propose a two-step domain-expert-Guided DG (GuiDG) framework. GuiDG first employs prompt tuning to obtain source domain experts, then introduces a Cross-Modal Attention module to guide the fine-tuning of the vision encoder via adaptive expert integration. To better evaluate few-shot DG, we construct ImageNet-DG from ImageNet and its variants. Extensive experiments on standard DG benchmarks and ImageNet-DG demonstrate that GuiDG improves upon state-of-the-art fine-tuning methods while maintaining efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。