arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.01917cs.CV
收藏

参数高效少样本微调的强基线

Strong Baselines for Parameter Efficient Few-Shot Fine-tuning

University of Maryland(马里兰大学);Microsoft Research(微软研究院);Samsung AI(三星人工智能)

Samyadeep Basu, Daniela Massiceti, Shell Xu Hu, Soheil Feizi

AI总结:

本文通过超过1.8k次受控实验分析了少样本分类中的参数高效微调方法,发现仅微调LayerNorm参数的LN-Tune是极强基线,而AttnScale结合DRA模块在自监督ViT上实现了最先进性能且参数效率提升约9倍。

摘要:

少样本分类(FSC)是指在预训练(或元训练)阶段基于一组基类进行学习后,每类仅给定少量样本来学习新类别。最近的研究表明,简单地在新测试类上微调预训练的 Vision Transformer (ViT) 是 FSC 的一种强大方法。然而,微调 ViT 在时间、计算和存储上代价高昂。这促使了参数高效微调(PEFT)方法的设计,该方法仅微调 Transformer 的一小部分参数。尽管这些方法显示出潜力,但实验条件的不一致性使得难以将其优势与其他实验因素(包括特征提取器架构、预训练初始化和微调算法等)区分开来。在本文中,我们进行了一项大规模、实验条件一致的实证分析,以研究少样本图像分类的 PEFT。通过在包括 Meta-Dataset (MD) 和 ORBIT 在内的大规模少样本基准上进行超过 1.8k 次受控实验,我们揭示了关于 PEFT 的新见解,阐明了其在为少样本分类微调 ViT 时的有效性。通过受控的实证研究,我们有两个主要发现:(i) 在少样本适应期间仅微调 LayerNorm 参数(我们称之为 LN-Tune)在自监督和监督目标预训练的 ViT 中都是极强的基线;(ii) 对于自监督 ViT,我们发现仅为每个注意力矩阵学习一组缩放参数(我们称之为 AttnScale)加上一个域残差适配器 (DRA) 模块,即可在 MD 上达到最先进的性能(同时参数效率提高约 9 倍)。我们广泛的实证发现设定了强基线,并呼吁重新思考当前 FSC 的 PEFT 方法设计。

英文摘要:

Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase on a set of base classes. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters. While these methods have shown promise, inconsistencies in experimental conditions make it difficult to disentangle their advantage from other experimental factors including the feature extractor architecture, pre-trained initialization and fine-tuning algorithm, amongst others. In our paper, we conduct a large-scale, experimentally consistent, empirical analysis to study PEFTs for few-shot image classification. Through a battery of over 1.8k controlled experiments on large-scale few-shot benchmarks including Meta-Dataset (MD) and ORBIT, we uncover novel insights on PEFTs that cast light on their efficacy in fine-tuning ViTs for few-shot classification. Through our controlled empirical study, we have two main findings: (i) Fine-tuning just the LayerNorm parameters (which we call LN-Tune) during few-shot adaptation is an extremely strong baseline across ViTs pre-trained with both self-supervised and supervised objectives, (ii) For self-supervised ViTs, we find that simply learning a set of scaling parameters for each attention matrix (which we call AttnScale) along with a domain-residual adapter (DRA) module leads to state-of-the-art performance (while being $\sim\!$ 9$\times$ more parameter-efficient) on MD. Our extensive empirical findings set strong baselines and call for rethinking the current design of PEFT methods for FSC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑