arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.00378cs.CL
收藏

参数高效微调中Transformer模块可移植性的实证研究

An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning

Sharif University of Technology(谢里夫理工大学);University of Amsterdam(阿姆斯特丹大学);Tehran Institute for Advanced Studies(德黑兰高等研究院)

Mohammad Akbar-Tajari, Sara Rajaee, Mohammad Taher Pilehvar

摘要:

参数高效微调方法最近引起了很多关注。这些方法具有相当低的可训练权重数,可以带来可扩展性和计算效率。在本文中,我们寻找最佳的子网络,并研究不同的Transformer模块在将知识从预先训练的模型转移到下游任务中的能力。我们的实证结果表明,BERT中的每个Transformer模块都可以作为获胜的门票:微调每个特定模块,同时保持网络的其余部分冻结,可以导致与完全微调相当的性能。在不同的模块中,LayerNorms在有限的可训练权重下表现出最好的知识转移能力,在某种程度上,在分层分析中只有0.003%的所有参数,它们在各种目标任务上表现出可接受的性能。关于其有效性背后的原因,我们认为,与预训练的BERT中的其他模块相比,其显着的性能可以归因于其高量级的权重。

英文摘要:

Parameter-efficient fine-tuning approaches have recently garnered a lot of attention. Having considerably lower number of trainable weights, these methods can bring about scalability and computational effectiveness. In this paper, we look for optimal sub-networks and investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. Our empirical results suggest that every transformer module in BERT can act as a winning ticket: fine-tuning each specific module while keeping the rest of the network frozen can lead to comparable performance to the full fine-tuning. Among different modules, LayerNorms exhibit the best capacity for knowledge transfer with limited trainable weights, to the extent that, with only 0.003% of all parameters in the layer-wise analysis, they show acceptable performance on various target tasks. On the reasons behind their effectiveness, we argue that their notable performance could be attributed to their high-magnitude weights compared to that of the other modules in the pre-trained BERT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑