arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.19286cs.CV
收藏

面向域泛化的软提示生成

Soft Prompt Generation for Domain Generalization

Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所);School of Electrical Engineering, Xi’an University of Technology(西安理工大学电气工程学院)

Shuanghao Bai, Yuedi Zhang, Wanqi Zhou, Zhirong Luan, Badong Chen

AI总结:

针对现有提示学习方法生成的提示缺乏多样性、忽略未见域信息的问题,本文提出软提示生成方法SPG,从生成视角重构提示学习框架,在多类域泛化基准上取得了当前最优性能。

摘要:

大型预训练视觉语言模型(VLM)在人工设计提示的下游任务中展现出出色的零样本能力。为进一步让VLM适配下游任务,学界提出软提示替代人工设计提示,软提示会基于特定域数据进行微调。此前的提示学习方法主要从训练样本中学习固定提示或残差提示,但所学习的提示缺乏多样性,且忽略了未见域的相关信息。本文从生成视角重构提示学习框架,针对域泛化(DG)任务提出一种简单高效的方法,即Soft Prompt Generation(SPG,软提示生成)。具体而言,SPG包含两阶段训练阶段和一个推理阶段:训练阶段为每个域引入软提示标签,旨在融入生成模型的域知识;推理阶段利用生成模型的生成器,为未见目标域生成实例专属的软提示。在三类DG任务的五个域泛化基准上开展的大量实验表明,SPG取得了当前最优性能。代码可在https://github.com/renytek13/Soft-Prompt-Generation-with-CGAN获取。

英文摘要:

Large pre-trained vision language models (VLMs) have shown impressive zero-shot ability on downstream tasks with manually designed prompt. To further adapt VLMs to downstream tasks, soft prompt is proposed to replace manually designed prompt, which undergoes fine-tuning based on specific domain data. Prior prompt learning methods primarily learn a fixed prompt or residuled prompt from training samples. However, the learned prompts lack diversity and ignore information about unseen domains. In this paper, we reframe the prompt learning framework from a generative perspective and propose a simple yet efficient method for the Domain Generalization (DG) task, namely Soft Prompt Generation (SPG). Specifically, SPG consists of a two-stage training phase and an inference phase. During the training phase, we introduce soft prompt label for each domain, aiming to incorporate the generative model domain knowledge. During the inference phase, the generator of the generative model is employed to obtain instance-specific soft prompts for the unseen target domain. Extensive experiments on five domain generalization benchmarks of three DG tasks demonstrate that SPG achieves state-of-the-art performance. The code is available at https://github.com/renytek13/Soft-Prompt-Generation-with-CGAN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑