如何准备您的任务头进行微调
How to prepare your task head for finetuning
摘要:
在深度学习中,通过微调将信息从预先训练的网络传输到下游任务有很多好处。任务头的选择在微调中起着重要作用,因为预训练和下游任务通常是不同的。尽管存在许多不同的微调设计,但对这些算法何时以及为什么工作的充分理解一直是难以捉摸的。我们分析了任务头的选择如何控制特征自适应,从而影响下游的性能。通过分解自适应的学习动态,我们发现关键方面是微调开始时的训练精度和损失,这决定了特征自适应的“能量”。我们确定了一个显着的趋势,在此初始能量的变化对微调后所产生的功能的影响。具体来说,随着能量的增加,结果和原始特征之间的欧几里得距离和余弦距离增加,而它们的点积(以及结果特征的范数)首先增加,然后减少。受此启发,我们给出了几个实用的原则,导致更好的下游性能。我们分析证明了这种趋势在一个overparameterized线性设置,并验证其适用于不同的实验设置。
英文摘要:
In deep learning, transferring information from a pretrained network to a downstream task by finetuning has many benefits. The choice of task head plays an important role in fine-tuning, as the pretrained and downstream tasks are usually different. Although there exist many different designs for finetuning, a full understanding of when and why these algorithms work has been elusive. We analyze how the choice of task head controls feature adaptation and hence influences the downstream performance. By decomposing the learning dynamics of adaptation, we find that the key aspect is the training accuracy and loss at the beginning of finetuning, which determines the "energy" available for the feature's adaptation. We identify a significant trend in the effect of changes in this initial energy on the resulting features after fine-tuning. Specifically, as the energy increases, the Euclidean and cosine distances between the resulting and original features increase, while their dot products (and the resulting features' norm) first increase and then decrease. Inspired by this, we give several practical principles that lead to better downstream performance. We analytically prove this trend in an overparamterized linear setting and verify its applicability to different experimental settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。