arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.08566cs.CVcs.AIcs.LG
收藏

敏感性感知的视觉参数高效微调

Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning

The University of Sydney(悉尼大学);Monash University(莫纳什大学)

Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, Bohan Zhuang

AI总结:

提出敏感性感知的视觉参数高效微调方案,自适应分配可训练参数到任务重要位置,结合结构化调优提升性能,在多个基准上达到最先进水平。

摘要:

视觉参数高效微调(PEFT)已成为全量微调的一种强大替代方案,用于将预训练的视觉模型适应到下游任务,它仅调整少量参数,同时冻结绝大多数参数,以减轻存储负担和优化难度。然而,现有的PEFT方法仅依赖人工启发式,在不同任务中向相同位置引入可训练参数,忽视了领域差异。为此,我们研究在哪里引入以及如何分配可训练参数,提出了一种新颖的敏感性感知的视觉参数高效微调(SPT)方案,该方案在给定所需可调参数预算的情况下,自适应地将可训练参数分配到任务特定的重要位置。具体来说,我们的SPT首先以数据依赖的方式快速识别给定任务中需要调优的敏感参数。接下来,对于敏感参数数量超过预定义阈值的权重矩阵,SPT利用现有的结构化调优方法(如LoRA [23]或Adapter [22])来增强其表示能力,以替代在预算下直接调优选定的敏感参数(非结构化调优)。在广泛的下游识别任务上进行的大量实验表明,我们的SPT与现有的PEFT方法互补,并大幅提升了它们的性能,例如,SPT将带有监督预训练ViT-B/16骨干的Adapter分别提高了4.2%和1.4%的平均Top-1准确率,在FGVC和VTAB-1k基准上达到了最先进的性能。源代码位于https://github.com/ziplab/SPT。

英文摘要:

Visual Parameter-Efficient Fine-Tuning (PEFT) has become a powerful alternative for full fine-tuning so as to adapt pre-trained vision models to downstream tasks, which only tunes a small number of parameters while freezing the vast majority ones to ease storage burden and optimization difficulty. However, existing PEFT methods introduce trainable parameters to the same positions across different tasks depending solely on human heuristics and neglect the domain gaps. To this end, we study where to introduce and how to allocate trainable parameters by proposing a novel Sensitivity-aware visual Parameter-efficient fine-Tuning (SPT) scheme, which adaptively allocates trainable parameters to task-specific important positions given a desired tunable parameter budget. Specifically, our SPT first quickly identifies the sensitive parameters that require tuning for a given task in a data-dependent way. Next, our SPT further boosts the representational capability for the weight matrices whose number of sensitive parameters exceeds a pre-defined threshold by utilizing existing structured tuning methods, e.g., LoRA [23] or Adapter [22], to replace directly tuning the selected sensitive parameters (unstructured tuning) under the budget. Extensive experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods and largely boosts their performance, e.g., SPT improves Adapter with supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean Top-1 accuracy, reaching SOTA performance on FGVC and VTAB-1k benchmarks, respectively. Source code is at https://github.com/ziplab/SPT

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑