视觉 Transformer 微调期间发生了什么：一项基于不变性的研究
What Happens During Finetuning of Vision Transformers: An Invariance Based Investigation
AI总结：
本文提出衡量不变性保留与遗忘的新指标，通过多个基准任务分析视觉 Transformer 微调前后的表征变化，揭示浅层可迁移不变性及深层不变性向浅层压缩的现象。
摘要：
预训练—微调范式通常会比在同一任务上从头训练模型带来更好的下游性能，因此已在机器学习的许多领域变得普遍。尽管从经验上观察到预训练对一系列任务有益，但人们尚未清楚理解产生这种效果的原因。在本工作中，我们在多个基准数据集和任务上考察预训练视觉 Transformer 与相应微调版本之间的关系。我们提出了新的指标，专门研究预训练模型所学到的不变性在微调过程中被保留或遗忘的程度。利用这些指标，我们给出了一组实证发现，包括预训练会在浅层诱导出可迁移的不变性，以及在微调过程中，来自更深层预训练层的不变性会被压缩到更浅的层。综合来看，这些发现有助于理解预训练模型取得成功的部分原因，以及预训练模型在下游任务上进行微调时所经历的变化。
英文摘要：
The pretrain-finetune paradigm usually improves downstream performance over training a model from scratch on the same task, becoming commonplace across many areas of machine learning. While pretraining is empirically observed to be beneficial for a range of tasks, there is not a clear understanding yet of the reasons for this effect. In this work, we examine the relationship between pretrained vision transformers and the corresponding finetuned versions on several benchmark datasets and tasks. We present new metrics that specifically investigate the degree to which invariances learned by a pretrained model are retained or forgotten during finetuning. Using these metrics, we present a suite of empirical findings, including that pretraining induces transferable invariances in shallow layers and that invariances from deeper pretrained layers are compressed towards shallower layers during finetuning. Together, these findings contribute to understanding some of the reasons for the successes of pretrained models and the changes that a pretrained model undergoes when finetuned on a downstream task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
