利用视觉Transformer基础模型中的层归一化微调进行分类
Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification
AI总结：
本文研究ViT中LayerNorm微调动态，提出FSR量化目标域代表性并设计标量λ重缩放机制与循环框架，有效解决数据稀缺与领域偏移下的微调问题。
摘要：
LayerNorm在Vision Transformers (ViTs)中至关重要，但其在数据稀缺和领域偏移下的微调动态仍未被充分探索。本文表明，微调后LayerNorm参数的偏移指示了源域和目标域之间的转换；其有效性取决于目标训练样本准确代表目标域的程度，这由我们提出的Fine-tuning Shift Ratio ($FSR$)量化。基于此，我们提出一种简单有效的重缩放机制，使用与$FSR$负相关的标量$λ$来对齐学习到的LayerNorm偏移与在完全代表性数据下实现的理想偏移，并结合循环框架进一步增强LayerNorm微调。在自然和病理图像上的广泛实验，涵盖分布内 (ID) 和分布外 (OOD) 设置及多种目标训练样本规模，验证了我们的框架。值得注意的是，与ID案例相比，OOD任务倾向于产生更低的$FSR$和更高的$λ$，特别是在数据稀缺时，表明目标训练样本代表性不足。此外，在病理数据上微调的ViTFs表现得更像ID设置，倾向于保守的LayerNorm更新。我们的发现阐明了迁移学习中LayerNorm未被充分探索的动态，并为LayerNorm微调提供了实用策略。
英文摘要：
LayerNorm is pivotal in Vision Transformers (ViTs), yet its fine-tuning dynamics under data scarcity and domain shifts remain underexplored. This paper shows that shifts in LayerNorm parameters after fine-tuning (LayerNorm shifts) are indicative of the transitions between source and target domains; its efficacy is contingent upon the degree to which the target training samples accurately represent the target domain, as quantified by our proposed Fine-tuning Shift Ratio ($FSR$). Building on this, we propose a simple yet effective rescaling mechanism using a scalar $λ$ that is negatively correlated to $FSR$ to align learned LayerNorm shifts with those ideal shifts achieved under fully representative data, combined with a cyclic framework that further enhances the LayerNorm fine-tuning. Extensive experiments across natural and pathological images, in both in-distribution (ID) and out-of-distribution (OOD) settings, and various target training sample regimes validate our framework. Notably, OOD tasks tend to yield lower $FSR$ and higher $λ$ in comparison to ID cases, especially with scarce data, indicating under-represented target training samples. Moreover, ViTFs fine-tuned on pathological data behave more like ID settings, favoring conservative LayerNorm updates. Our findings illuminate the underexplored dynamics of LayerNorm in transfer learning and provide practical strategies for LayerNorm fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
