arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.05067cs.LGcs.AIcs.CV
收藏

改进自监督视觉Transformer的视觉提示调优

Improving Visual Prompt Tuning for Self-supervised Vision Transformers

Seoul National University(首尔国立大学)

Seungryong Yoo, Eunji Kim, Dahuin Jung, Jungbeom Lee, Sungroh Yoon

AI总结:

针对视觉提示调优在自监督视觉Transformer上表现不佳的问题,提出为每个ViT块学习门控以选择性干预提示令牌的方法,在多项图像分类和语义分割任务上优于现有VPT变体。

摘要:

视觉提示调优(Visual Prompt Tuning, VPT)是一种将预训练视觉Transformer(Vision Transformers, ViTs)适配到下游任务的有效调优方法。它利用额外的可学习令牌(即提示)来引导冻结的预训练ViTs。尽管VPT已在监督视觉Transformer上证明了适用性,但在自监督视觉Transformer上往往表现不佳。通过实验观察,我们推断VPT的有效性很大程度上取决于提示令牌与之交互的ViT块。具体而言,当提示令牌被插入到后几个块而非第一个块时,VPT在MAE和MoCo v3的图像分类任务上表现出提升的性能。这些观察结果表明,存在一个插入提示令牌的最优块位置。遗憾的是,针对不同的未来场景,在每个自监督ViT中确定提示的最优块是一个成本高昂的过程。为缓解这一问题,我们提出了一种简单而有效的方法:为每个ViT块学习一个门控,以调整其对提示令牌的干预程度。通过我们的方法,提示令牌会选择性地受到那些任务适配需要引导的块的影响。我们的方法在FGVC和VTAB图像分类以及ADE20K语义分割任务上优于各类VPT变体。代码可在https://github.com/ryongithub/GatedPromptTuning获取。

英文摘要:

Visual Prompt Tuning (VPT) is an effective tuning method for adapting pretrained Vision Transformers (ViTs) to downstream tasks. It leverages extra learnable tokens, known as prompts, which steer the frozen pretrained ViTs. Although VPT has demonstrated its applicability with supervised vision transformers, it often underperforms with self-supervised ones. Through empirical observations, we deduce that the effectiveness of VPT hinges largely on the ViT blocks with which the prompt tokens interact. Specifically, VPT shows improved performance on image classification tasks for MAE and MoCo v3 when the prompt tokens are inserted into later blocks rather than the first block. These observations suggest that there exists an optimal location of blocks for the insertion of prompt tokens. Unfortunately, identifying the optimal blocks for prompts within each self-supervised ViT for diverse future scenarios is a costly process. To mitigate this problem, we propose a simple yet effective method that learns a gate for each ViT block to adjust its intervention into the prompt tokens. With our method, prompt tokens are selectively influenced by blocks that require steering for task adaptation. Our method outperforms VPT variants in FGVC and VTAB image classification and ADE20K semantic segmentation. The code is available at https://github.com/ryongithub/GatedPromptTuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑