arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.10973cs.CVcs.AI
收藏

ExPLoRA: 低参数高效扩展预训练以适应视觉变换器下的领域偏移

ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

stanford(斯坦福大学)

Samar Khanna, Medhanie Irgau, David B. Lobell, Stefano Ermon

AI总结:

ExPLoRA通过扩展预训练阶段并结合LoRA技术,在领域偏移下提升视觉变换器的迁移学习性能。

摘要:

参数高效的微调(PEFT)技术,如低秩适应(LoRA),可以使用仅原始可训练权重的小部分(0.1%-10%)有效地将大预训练基础模型适应到下游任务。PEFT中一个未被充分探索的问题是扩展预训练阶段而不使用监督标签;也就是说,我们能否通过在该领域上高效的自监督预训练来适应预训练的基础模型?在本工作中,我们介绍了ExPLoRA,一种高度有效的技术,用于改进预训练视觉变换器(ViTs)在领域偏移下的迁移学习。初始化一个ViT,使用在大型自然图像数据集(如DinoV2或MAE)上的预训练权重,ExPLoRA继续在新领域上进行无监督的预训练目标,解冻1-2个预训练的ViT块,并使用LoRA调节其他所有层。然后,我们仅在该新领域上使用LoRA对模型进行微调以进行监督学习。我们的实验显示在卫星图像上取得了最先进的结果,甚至优于完全预训练和微调的ViTs。使用DinoV2训练目标,我们在下游任务上展示了线性探测top-1准确率的最高提升达8%,同时使用的参数数量少于先前完全微调的最先进方法的10%。我们的消融研究证实了我们的方法在与其他基线(如PEFT)相比的有效性。代码可在项目网站上获得:https://samar-khanna.github.io/ExPLoRA/

英文摘要:

Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effectively adapt large pre-trained foundation models to downstream tasks using only a small fraction (0.1%-10%) of the original trainable weights. An under-explored question of PEFT is in extending the pre-training phase without supervised labels; that is, can we adapt a pre-trained foundation model to a new domain via efficient self-supervised pre-training on this domain? In this work, we introduce ExPLoRA, a highly effective technique to improve transfer learning of pre-trained vision transformers (ViTs) under domain shifts. Initializing a ViT with pre-trained weights on large, natural-image datasets such as from DinoV2 or MAE, ExPLoRA continues the unsupervised pre-training objective on a new domain, unfreezing 1-2 pre-trained ViT blocks and tuning all other layers with LoRA. We then fine-tune the resulting model only with LoRA on this new domain for supervised learning. Our experiments demonstrate state-of-the-art results on satellite imagery, even outperforming fully pre-training and fine-tuning ViTs. Using the DinoV2 training objective, we demonstrate up to 8% improvement in linear probing top-1 accuracy on downstream tasks while using <10% of the number of parameters that are used in prior fully-tuned state-of-the-art approaches. Our ablation studies confirm the efficacy of our approach over other baselines such as PEFT. Code is available on the project website: https://samar-khanna.github.io/ExPLoRA/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑