arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.09792cs.CV
收藏

面向域自适应稠密预测的稀疏视觉提示探索

Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction

Peking University(北京大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));University of Science and Technology of China(中国科学技术大学);Beihang University(北京航空航天大学)

Senqiao Yang, Jiarui Wu, Jiaming Liu, Xiaoqi Li, Qizhe Zhang, Mingjie Pan, Yulu Gan, Zehui Chen, Shanghang Zhang

AI总结:

针对现有图像级视觉域提示处理稠密预测测试时自适应任务时存在空间细节丢失等问题,提出稀疏视觉域提示方法及配套的提示布局与更新策略,仅用极少参数就实现了语义分割和深度估计任务的SOTA性能。

摘要:

视觉提示为解决视觉跨域问题提供了一种高效方式。在以往研究中,Visual Domain Prompt(VDP,视觉域提示)首次引入域提示来处理分类任务的Test-Time Adaptation(TTA,测试时自适应)问题,其方式是在输入图像上叠加图像级提示,并针对每个目标域微调提示。但由于图像级提示会遮蔽提示分配区域内的连续空间细节,因此会存在上下文信息不准确、域知识提取受限的问题,在处理稠密预测TTA问题时尤为明显。为克服这些挑战,我们提出了一种全新的Sparse Visual Domain Prompts(SVDP,稀疏视觉域提示)方法,该方法在图像级提示中仅保留极少的可训练参数(例如0.1%),从而保留输入的更多空间信息。为了更好地利用SVDP提取域专属知识,我们提出了Domain Prompt Placement(DPP,域提示布局)方法,可将SVDP的可训练参数自适应地分配到分布偏移较大的像素上。此外,考虑到每个目标域样本都存在独特的域偏移,我们设计了Domain Prompt Updating(DPU,域提示更新)策略,针对每个样本差异化地优化提示参数,助力高效适配目标域。我们在广泛使用的TTA基准和持续TTA基准上开展了大量实验,所提方法在语义分割和深度估计任务中均取得了当前最优性能。

英文摘要:

The visual prompts have provided an efficient manner in addressing visual cross-domain problems. In previous works, Visual Domain Prompt (VDP) first introduces domain prompts to tackle the classification Test-Time Adaptation (TTA) problem by warping image-level prompts on the input and fine-tuning prompts for each target domain. However, since the image-level prompts mask out continuous spatial details in the prompt-allocated region, it will suffer from inaccurate contextual information and limited domain knowledge extraction, particularly when dealing with dense prediction TTA problems. To overcome these challenges, we propose a novel Sparse Visual Domain Prompts (SVDP) approach, which holds minimal trainable parameters (e.g., 0.1\%) in the image-level prompt and reserves more spatial information of the input. To better apply SVDP in extracting domain-specific knowledge, we introduce the Domain Prompt Placement (DPP) method to adaptively allocates trainable parameters of SVDP on the pixels with large distribution shifts. Furthermore, recognizing that each target domain sample exhibits a unique domain shift, we design Domain Prompt Updating (DPU) strategy to optimize prompt parameters differently for each sample, facilitating efficient adaptation to the target domain. Extensive experiments were conducted on widely-used TTA and continual TTA benchmarks, and our proposed method achieves state-of-the-art performance in both semantic segmentation and depth estimation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑