Vision Mamba中的选择性视觉提示
Selective Visual Prompting in Vision Mamba
AI总结：
针对现有视觉提示方法不适配Vision Mamba特性的问题，提出带双路径结构的选择性视觉提示方法，通过逐token提示激活门控促进信息传播，在多个大规模基准上性能显著优于现有最优方法。
摘要：
预训练Vision Mamba（Vim，视觉曼巴）模型凭借其选择性状态空间模型的独特设计，已在各类计算机视觉任务中以高效计算的方式展现出卓越性能。为进一步将其适用性拓展至多样的下游视觉任务，可采用视觉提示这一高效微调技术适配Vim模型。然而，现有视觉提示方法主要针对利用全局注意力的Vision Transformer（ViT，视觉Transformer）模型定制，忽略了Vim独特的序列逐token压缩与传播特性。具体而言，现有附加在序列前端的提示token不足以有效激活整个序列的输入门与遗忘门，阻碍了判别性信息的提取与传播。为解决这一局限，我们提出了一种专门用于Vim高效微调的新型选择性视觉提示（Selective Visual Prompting, SVP）方法。为防止状态空间传播过程中判别性信息的丢失，SVP采用轻量级选择性提示器生成逐token提示，确保自适应激活Mamba块内的更新门与遗忘门，以促进判别性信息的传播。此外，考虑到Vim同时传播跨层共享信息和层内特定信息，我们进一步以双路径结构优化SVP：跨提示（Cross-Prompting）与层内提示（Inner-Prompting）。跨提示在层间使用共享参数，层内提示则采用不同参数，分别促进共享信息与特定信息的传播。在多个大规模基准上的广泛实验结果表明，我们提出的SVP显著优于当前最优方法。我们的代码可在https://github.com/zhoujiahuan1991/AAAI2025-SVP获取。
英文摘要：
Pre-trained Vision Mamba (Vim) models have demonstrated exceptional performance across various computer vision tasks in a computationally efficient manner, attributed to their unique design of selective state space models. To further extend their applicability to diverse downstream vision tasks, Vim models can be adapted using the efficient fine-tuning technique known as visual prompting. However, existing visual prompting methods are predominantly tailored for Vision Transformer (ViT)-based models that leverage global attention, neglecting the distinctive sequential token-wise compression and propagation characteristics of Vim. Specifically, existing prompt tokens prefixed to the sequence are insufficient to effectively activate the input and forget gates across the entire sequence, hindering the extraction and propagation of discriminative information. To address this limitation, we introduce a novel Selective Visual Prompting (SVP) method specifically for the efficient fine-tuning of Vim. To prevent the loss of discriminative information during state space propagation, SVP employs lightweight selective prompters for token-wise prompt generation, ensuring adaptive activation of the update and forget gates within Mamba blocks to promote discriminative information propagation. Moreover, considering that Vim propagates both shared cross-layer information and specific inner-layer information, we further refine SVP with a dual-path structure: Cross-Prompting and Inner-Prompting. Cross-Prompting utilizes shared parameters across layers, while Inner-Prompting employs distinct parameters, promoting the propagation of both shared and specific information, respectively. Extensive experimental results on various large-scale benchmarks demonstrate that our proposed SVP significantly outperforms state-of-the-art methods. Our code is available at https://github.com/zhoujiahuan1991/AAAI2025-SVP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
