FullLoRA: Efficiently Boosting the Robustness of Pretrained Vision Transformers
Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing 100190, China, and the University of Chinese Academy of Sciences, Beijing 100049, China（中国科学院计算技术研究所人工智能安全重点实验室，中国科学院（CAS）及中国科学院大学，北京）
Zheng Yuan, Jie Zhang, Shiguang Shan, Xilin Chen
备注：
Accepted by IEEE Transactions on Image Processing (TIP). 11 pages, 3 figures, 8 tables
In recent years, the Vision Transformer (ViT) model has gradually become mainstream in various computer vision tasks, and the robustness of the model has received increasing attention. However, existing large models tend to prioritize performance during training, potentially neglecting the robustness, which may lead to serious security concerns. In this paper, we establish a new challenge: exploring how to use a small number of additional parameters for adversarial finetuning to quickly and effectively enhance the adversarial robustness of a standardly trained model. To address this challenge, we develop novel LNLoRA module, incorporating a learnable layer normalization before the conventional LoRA module, which helps mitigate magnitude differences in parameters between the adversarial and standard training paradigms. Furthermore, we propose the FullLoRA framework by integrating the learnable LNLoRA modules into all key components of ViT-based models while keeping the pretrained model frozen, which can significantly improve the model robustness via adversarial finetuning in a parameter-efficient manner. Extensive experiments on several datasets demonstrate the superiority of our proposed FullLoRA framework. It achieves comparable robustness with full finetuning while only requiring about 5\% of the learnable parameters. This also effectively addresses concerns regarding extra model storage space and enormous training time caused by adversarial finetuning.
Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models
Zhejiang University（浙江大学）；Shanghai Maritime University（上海海事大学）；Zhejiang Gongshang University（浙江工商大学）；Nanjing University of Information Science and Technology（南京信息工程大学）
University of Luxembourg（卢森堡大学）；University of California, Berkeley（加州大学伯克利分校）
We propose transferability from Large Geometric Vicinity (LGV), a new technique to increase the transferability of black-box adversarial attacks. LGV starts from a pretrained surrogate model and collects multiple weight sets from a few additional training epochs with a constant and high learning rate. LGV exploits two geometric properties that we relate to transferability. First, models that belong to a wider weight optimum are better surrogates. Second, we identify a subspace able to generate an effective surrogate ensemble among this wider optimum. Through extensive experiments, we show that…
Texas A&M University（得克萨斯农工大学）；MIT-IBM Watson AI Lab, IBM Research（麻省理工学院-IBM沃森人工智能实验室，IBM研究院）；Microsoft Dynamics 365 AI Research（微软Dynamics 365人工智能研究院）
Pretrained models from self-supervision are prevalently used in fine-tuning downstream tasks faster or for better accuracy. However, gaining robustness from pretraining is left unexplored. We introduce adversarial training into self-supervision, to provide general-purpose robust pre-trained models for the first time. We find these robust pre-trained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine-tuning. We conduct extensive experiments to demonstrate that the proposed framewo…
Dalian University of Technology（大连理工大学）；Peng Cheng Laboratory（鹏城实验室）；Remark AI
Visual object tracking aims to precisely estimate the bounding box for the given target, which is a challenging problem due to factors such as deformation and occlusion. Many recent trackers adopt the multiple-stage tracking strategy to improve the quality of bounding box estimation. These methods first coarsely locate the target and then refine the initial prediction in the following stages. However, existing approaches still suffer from limited precision, and the coupling of different stages severely restricts the method's transferability. This work proposes a novel, flexible, and accurate r…
Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（信息科技学院，科学大学，胡志明市，越南）；Vietnam National University, Ho Chi Minh City, Vietnam（越南国家大学，胡志明市，越南）
