arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.02797cs.CVcs.LG
收藏

通过视觉域提示生成适应分布偏移

Adapting to Distribution Shift by Visual Domain Prompt Generation

University of Toronto(多伦多大学);Concordia University(康考迪亚大学);McMaster University(麦克马斯特大学)

Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, Yuanhao Yu, Konstantinos N Plataniotis, Yang Wang

AI总结:

本文提出基于基础模型预计算特征构建知识库,并通过域提示生成器将源域知识压缩为域特定提示,结合域感知对比损失与元学习,在测试时利用少量无标签数据适应分布偏移,在多个大规模基准上取得领先性能。

摘要:

在本文中,我们旨在利用少量无标签数据在测试时对模型进行适应,以应对分布偏移。为了解决从有限数据中提取域知识的挑战,利用来自预训练骨干网络和源域的相关信息至关重要。以往的研究未能利用具有强分布外泛化能力的近期基础模型。此外,以域为中心的设计在其工作中未得到重视。而且,它们将源域建模过程和适应学习过程分别置于互不相干的训练阶段。在本工作中,我们提出一种基于基础模型预计算特征的方法。具体来说,我们构建一个知识库,从源域中学习可迁移的知识。以少样本目标数据为条件,我们引入一个域提示生成器,将知识库压缩为域特定的提示。随后,该域提示通过一个引导模块将视觉特征导向特定域。此外,我们提出一种域感知对比损失,并采用元学习来促进域知识提取。我们进行了大量实验以验证域知识提取的有效性。所提方法在包括WILDS和DomainNet在内的5个大规模基准上优于先前工作。

英文摘要:

In this paper, we aim to adapt a model at test-time using a few unlabeled data to address distribution shifts. To tackle the challenges of extracting domain knowledge from a limited amount of data, it is crucial to utilize correlated information from pre-trained backbones and source domains. Previous studies fail to utilize recent foundation models with strong out-of-distribution generalization. Additionally, domain-centric designs are not flavored in their works. Furthermore, they employ the process of modelling source domains and the process of learning to adapt independently into disjoint training stages. In this work, we propose an approach on top of the pre-computed features of the foundation model. Specifically, we build a knowledge bank to learn the transferable knowledge from source domains. Conditioned on few-shot target data, we introduce a domain prompt generator to condense the knowledge bank into a domain-specific prompt. The domain prompt then directs the visual features towards a particular domain via a guidance module. Moreover, we propose a domain-aware contrastive loss and employ meta-learning to facilitate domain knowledge extraction. Extensive experiments are conducted to validate the domain knowledge extraction. The proposed method outperforms previous work on 5 large-scale benchmarks including WILDS and DomainNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑