arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.13982cs.CVcs.LG
收藏

基于属性的视觉重编程用于视觉-语言模型

Attribute-based Visual Reprogramming for Vision-Language Models

The University of Melbourne(墨尔本大学);Southeast University(东南大学);Idealism Technology (Beijing)(北京理想主义科技公司)

Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, Feng Liu

AI总结:

提出基于属性的视觉重编程方法AttrVR,利用描述性和区分性属性迭代优化噪声模式,减少类内方差、增加类间分离,在12个下游任务上超越现有方法。

摘要:

视觉重编程(VR)通过向输入添加可训练的噪声模式,重用预训练的视觉模型进行下游图像分类任务。当应用于视觉-语言模型(如CLIP)时,现有的VR方法遵循与视觉模型(如ResNet、ViT)相同的流程,即将真实类别标签插入固定的文本模板中以指导VR模式的优化。然而,这种基于标签的方法忽视了CLIP可以利用的丰富信息和多样化的属性引导的文本表示,这可能导致样本的错误分类。在本文中,我们提出了用于CLIP的基于属性的视觉重编程(AttrVR),利用描述性属性(DesAttrs)和区分性属性(DistAttrs),它们分别代表不同类别的共同和独特的特征描述。此外,由于同一类别的图像在VR后可能反映不同的属性,AttrVR使用每个图像样本的$k$-最近DesAttrs和DistAttrs迭代地细化模式,从而实现更动态和样本特定的优化。理论上,AttrVR被证明可以减少类内方差并增加类间分离。实验上,它在基于ViT和基于ResNet的CLIP的12个下游任务中取得了优越的性能。AttrVR的成功促进了VR从单模态视觉模型更有效地集成到视觉-语言模型中。我们的代码可在https://github.com/tmlr-group/AttrVR获取。

英文摘要:

Visual reprogramming (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose Attribute-based Visual Reprogramming (AttrVR) for CLIP, utilizing descriptive attributes (DesAttrs) and distinctive attributes (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the $k$-nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models. Our code is available at https://github.com/tmlr-group/AttrVR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑