arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06883cs.LGcs.CVstat.ML
收藏

视觉变换器微调受益于非平滑组件

Vision Transformer Finetuning Benefits from Non-Smooth Components

Noah's Ark Lab(诺亚 ark 实验室);Univ. Rennes 2, Inria(里昂二大学,法国国家信息与自动化研究所)

Ambroise Odonnat, Laetitia Chapel, Romain Tavenard, Ievgen Redko

AI总结:

本文通过分析视觉变换器组件的可塑性(即输出对输入变化的敏感度),发现高可塑性(低平滑性)的注意力模块和前馈层在微调中表现更好,挑战了平滑性有利的传统观点。

摘要:

变换器架构的平滑性在泛化、训练稳定性和对抗鲁棒性方面已被广泛研究。然而,其在迁移学习中的作用仍知之甚少。本文分析了视觉变换器组件使其输出适应输入变化的能力,即它们的\emph{可塑性}。定义为平均变化率,它捕捉了对输入扰动的敏感性;特别地,高可塑性意味着低平滑性。我们的理论分析和大量实验——在大规模视觉变换器上进行超过1000次微调运行——表明,这一视角为选择在适应过程中优先考虑的组件提供了原则性指导。对从业者的关键启示是,注意力模块和前馈层的高可塑性始终导致更好的微调性能。我们的发现偏离了平滑性是可取的普遍假设,为变换器的功能特性提供了新的视角。代码可在 https://github.com/ambroiseodt/vit-plasticity 获取。

英文摘要:

The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their \emph{plasticity}. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies a low smoothness. Our theoretical analysis and extensive experiments -- over $1,000$ finetuning runs on large-scale vision transformers -- showcase that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on transformers' functional properties. The code is available at https://github.com/ambroiseodt/vit-plasticity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑