对视觉与语言多提示学习的回顾
A Retrospect to Multi-prompt Learning across Vision and Language
AI总结：
本文回顾视觉语言多提示学习，提出基于能量的多提示学习方法（EMPL），通过从VLMs隐式定义的能量分布生成多个提示嵌入，在参数高效的同时实现域内与域外开放词汇泛化的平衡。
摘要：
随着视觉语言预训练模型（VLMs）的出现，视觉领域正经历前所未有的进展。提示学习作为访问VLMs的圣杯，使其能够以有限资源快速适应下游任务。然而现有研究多围绕单提示范式，很少探究多提示学习对应方法的技术潜力。本文旨在为视觉语言多提示学习提供有原则的回顾。我们将近期的恒定模态鸿沟现象扩展到可学习提示，并从经验和理论上证明多提示增强在视觉语言迁移中的优越性。基于此观察，我们提出基于能量的多提示学习（EMPL），通过从VLMs隐式定义的基于能量的分布中抽取实例来生成多个提示嵌入。因此，EMPL不仅参数高效，还能严格实现域内与域外开放词汇泛化的平衡。全面实验验证了我们的主张和EMPL的卓越性。
英文摘要：
The vision community is undergoing the unprecedented progress with the emergence of Vision-Language Pretraining Models (VLMs). Prompt learning plays as the holy grail of accessing VLMs since it enables their fast adaptation to downstream tasks with limited resources. Whereas existing researches milling around single-prompt paradigms, rarely investigate the technical potential behind their multi-prompt learning counterparts. This paper aims to provide a principled retrospect for vision-language multi-prompt learning. We extend the recent constant modality gap phenomenon to learnable prompts and then, justify the superiority of vision-language transfer with multi-prompt augmentation, empirically and theoretically. In terms of this observation, we propose an Energy-based Multi-prompt Learning (EMPL) to generate multiple prompt embeddings by drawing instances from an energy-based distribution, which is implicitly defined by VLMs. So our EMPL is not only parameter-efficient but also rigorously lead to the balance between in-domain and out-of-domain open-vocabulary generalization. Comprehensive experiments have been conducted to justify our claims and the excellence of EMPL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
