arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.03150cs.LGcs.CV
收藏

基于视觉重编程提示的样本特定掩码

Sample-specific Masks for Visual Reprogramming-based Prompting

The University of Melbourne(墨尔本大学);Singapore University of Technology and Design(新加坡科技设计大学)

Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, Feng Liu

AI总结:

针对视觉重编程中共享掩码限制泛化并增加近似误差的问题,提出样本特定多通道掩码(SMM)框架,利用轻量级ConvNet生成特定掩码,有效降低近似误差并在ResNet和ViT上取得性能提升。

摘要:

视觉重编程(VR)是一种提示技术,旨在通过学习添加到输入图像中的小规模模式,而非调整模型内部的大量参数,将预训练模型(例如 ImageNet 上的分类器)重新用于目标任务(例如医学数据预测)。输入样本中模式的位置通常由所有样本共享的预定义掩码决定。在本文中,我们表明,由于缺乏样本级适应性,共享掩码潜在地限制了 VR 的泛化能力并增加了其近似误差。受此发现的启发,我们为 VR 设计了一个名为样本特定多通道掩码(SMM)的新框架。具体而言,SMM 采用轻量级 ConvNet 和逐块插值来生成样本特定的三通道掩码,而不是共享的预定义掩码。由于我们为各个样本生成不同的掩码,理论上证明,与现有最先进的 VR 方法相比,SMM 能够减少目标任务的近似误差。我们还在 ResNet 和 ViT 上实证证明了其性能提升。SMM 的成功进一步突显了 VR 在利用预训练模型潜在知识处理各种目标任务方面的更广泛适用性。我们的代码可在 https://github.com/tmlr-group/SMM 获取。

英文摘要:

Visual reprogramming (VR) is a prompting technique that aims to re-purpose a pre-trained model (e.g., a classifier on ImageNet) to target tasks (e.g., medical data prediction) by learning a small-scale pattern added into input images instead of tuning considerable parameters within the model. The location of the pattern within input samples is usually determined by a pre-defined mask shared across all samples. In this paper, we show that the shared mask potentially limits VR's generalization and increases its approximation error due to the lack of sample-level adaptation. Motivated by this finding, we design a new framework for VR called sample-specific multi-channel masks (SMM). Specifically, SMM employs a lightweight ConvNet and patch-wise interpolation to generate sample-specific three-channel masks instead of a shared and pre-defined mask. Since we generate different masks for individual samples, SMM is theoretically shown to reduce approximation error for the target tasks compared with existing state-of-the-art VR methods. We also empirically demonstrate its performance gain on both ResNet and ViT. The success of SMM further highlights the broader applicability of VR in leveraging the latent knowledge of pre-trained models for various target tasks. Our code is available at https://github.com/tmlr-group/SMM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑