arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.14102cs.CLcs.AI
收藏

探索预训练语言模型的模式连接性

Exploring Mode Connectivity for Pre-trained Language Models

Tsinghua University(清华大学);Renmin University of China(中国人民大学);International Innovation Center of Tsinghua University(清华大学国际创新中心);Tencent Inc.(腾讯公司);Quan Cheng Laboratory(泉城实验室)

Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, Jie Zhou

摘要:

近年来,预训练的语言模型(PLMs)在自然语言处理中得到了广泛的应用。从参数空间的角度来看,PLM提供了通用的初始化,从初始化开始可以找到高性能的最小值,虽然大量的工作已经研究了如何有效地和高效地使PLM适应高性能的最小值,但很少有人知道在不同的自适应配置下达到的各种最小值之间的联系。本文从模式连通性的角度研究了两个极小点之间的几何连接,并通过实证分析探讨了三个问题:(1)超参数、特定的调谐方法和训练数据如何影响PLM的模式连通性?(2)在预训练期间,模式连接性如何变化?(3)PLM的任务知识如何沿着连接两个极小值的路径变化?一般而言,探索PLM的模式连接有助于理解不同极小值的几何连接,这可能有助于我们深入了解PLM下游适应的内部工作原理。

英文摘要:

Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found. Although plenty of works have studied how to effectively and efficiently adapt PLMs to high-performance minima, little is known about the connection of various minima reached under different adaptation configurations. In this paper, we investigate the geometric connections of different minima through the lens of mode connectivity, which measures whether two minima can be connected with a low-loss path. We conduct empirical analyses to investigate three questions: (1) how could hyperparameters, specific tuning methods, and training data affect PLM's mode connectivity? (2) How does mode connectivity change during pre-training? (3) How does the PLM's task knowledge change along the path connecting two minima? In general, exploring the mode connectivity of PLMs conduces to understanding the geometric connection of different minima, which may help us fathom the inner workings of PLM downstream adaptation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑