arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17406cs.CVcs.AIcs.LG
收藏

LSPT:面向视觉表示学习的长期空间提示微调

LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning

Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Microsoft Research(微软研究院)

Shentong Mo, Yansen Wang, Xufang Luo, Dongsheng Li

AI总结:

本文提出长期空间提示微调(LSPT),通过长期门控提示和图像块令牌分别实现时间与空间编码,以缓解自监督ViT中的参数遗忘并增强类别特征积累,在多个FGVC和VTAB-1K基准上取得领先性能。

摘要:

视觉提示微调(VPT)技术因其能够利用被称为提示(prompts)的专用可学习令牌,将预训练的视觉Transformer(ViTs)适配到下游视觉任务而受到广泛关注。当前的VPT方法,尤其是在与自监督视觉Transformer结合使用时,通常默认引入新的可学习提示或主要来源于模型前一个块的门控提示令牌。这类方法的一个关键疏漏在于,它们未能利用每个自监督ViT内部更早的、长距离的前序块作为提示来源。为弥补这一重要空白,我们提出了长期空间提示微调(LSPT)——一种面向视觉表示学习的革新性方法。受人类大脑复杂机制的启发,LSPT巧妙地引入了长期门控提示。这一特性充当时间编码,降低了遗忘从更早块中获取的参数的风险。为进一步增强其能力,LSPT引入了图像块令牌,充当空间编码。这一设计策略性地持续积累类别感知特征,从而增强模型在区分和识别视觉类别方面的能力。为验证所提方法的有效性,我们在5个FGVC和19个VTAB-1K基准上进行了严格的实验。我们的实证结果凸显了LSPT的优越性,表明其能够在视觉提示微调性能方面树立新的基准。

英文摘要:

Visual Prompt Tuning (VPT) techniques have gained prominence for their capacity to adapt pre-trained Vision Transformers (ViTs) to downstream visual tasks using specialized learnable tokens termed as prompts. Contemporary VPT methodologies, especially when employed with self-supervised vision transformers, often default to the introduction of new learnable prompts or gated prompt tokens predominantly sourced from the model's previous block. A pivotal oversight in such approaches is their failure to harness the potential of long-range previous blocks as sources of prompts within each self-supervised ViT. To bridge this crucial gap, we introduce Long-term Spatial Prompt Tuning (LSPT) - a revolutionary approach to visual representation learning. Drawing inspiration from the intricacies of the human brain, LSPT ingeniously incorporates long-term gated prompts. This feature serves as temporal coding, curbing the risk of forgetting parameters acquired from earlier blocks. Further enhancing its prowess, LSPT brings into play patch tokens, serving as spatial coding. This is strategically designed to perpetually amass class-conscious features, thereby fortifying the model's prowess in distinguishing and identifying visual categories. To validate the efficacy of our proposed method, we engaged in rigorous experimentation across 5 FGVC and 19 VTAB-1K benchmarks. Our empirical findings underscore the superiority of LSPT, showcasing its ability to set new benchmarks in visual prompt tuning performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑