arXiv 2410.059512024-10-08cs.CV

用于提升预训练大视觉模型对抗鲁棒性的超对抗调优

Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models

Zhejiang University（浙江大学）；Shanghai Maritime University（上海海事大学）；Zhejiang Gongshang University（浙江工商大学）；Nanjing University of Information Science and Technology（南京信息工程大学）

Kangtao Lv, Huangsen Cao, Kainan Tu, Yihuai Xu, Zhimeng Zhang, Xin Ding, Yongwei Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对大视觉模型对抗训练成本高、现有鲁棒微调方法效果不及全参数微调的问题，提出超对抗调优方法HyperAT，通过超网络生成不同防御的LoRA并融合，高效提升模型对抗鲁棒性，达到新的SOTA。

摘要：

大视觉模型已被发现易受对抗样本攻击，凸显了提升其对抗鲁棒性的必要性。对抗训练虽是深度卷积模型的有效防御手段，但因计算成本高，在大视觉模型上常面临可扩展性问题。近期研究提出了鲁棒微调方法，例如在大视觉模型中对低秩适配（LoRA）进行对抗调优，但这类方法的准确率仍难以匹敌全参数对抗微调。整合多种防御机制为提升大视觉模型鲁棒性提供了可行思路，但该范式目前仍未得到充分探索。为解决这一问题，我们提出超对抗调优（HyperAT），该方法利用不同防御方法间的共享防御知识，同时高效且有效地提升模型鲁棒性。具体而言，我们将每种防御方法的对抗调优建模为一个学习任务，由一个超网络生成适配该防御的专属LoRA。随后，我们提出一种随机采样与调优策略，以提取并促进不同防御方法之间的防御知识迁移。最后，将多样化的LoRA进行融合以增强对抗鲁棒性。在多个数据集和模型架构上的实验表明，HyperAT在不产生过高计算开销的情况下，显著提升了预训练大视觉模型的对抗鲁棒性，树立了新的 state-of-the-art 基准。

英文摘要：

Large vision models have been found vulnerable to adversarial examples, emphasizing the need for enhancing their adversarial robustness. While adversarial training is an effective defense for deep convolutional models, it often faces scalability issues with large vision models due to high computational costs. Recent approaches propose robust fine-tuning methods, such as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but they still struggle to match the accuracy of full parameter adversarial fine-tuning. The integration of various defense mechanisms offers a promising approach to enhancing the robustness of large vision models, yet this paradigm remains underexplored. To address this, we propose hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously. Specifically, adversarial tuning of each defense method is formulated as a learning task, and a hypernetwork generates LoRA specific to this defense. Then, a random sampling and tuning strategy is proposed to extract and facilitate the defensive knowledge transfer between different defenses. Finally, diverse LoRAs are merged to enhance the adversarial robustness. Experiments on various datasets and model architectures demonstrate that HyperAT significantly enhances the adversarial robustness of pretrained large vision models without excessive computational overhead, establishing a new state-of-the-art benchmark.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。