Transformer学习动力学中的分布性简单性偏差
A distributional simplicity bias in the learning dynamics of transformers
AI总结:
本研究发现自监督训练的Transformer存在分布性简单性偏差,会按序学习词元多体相互作用,低阶先饱和,同时提出数据集克隆方法以支撑该分析,拓展了相关研究路径。
摘要:
过参数化神经网络出色的泛化能力,一直以来被认为源于“简单性偏差”:神经网络会先学习简单分类器,再逐步学习更复杂的非线性函数,以此避免过拟合。尽管简单性偏差已在前馈网络的监督学习场景中得到理论和实验层面的描述,但它在多大程度上能解释经自监督技术训练的Transformer所取得的显著成功,目前仍不明确。本研究证明,在自然语言数据上训练的Transformer同样存在简单性偏差。具体而言,它们会按顺序学习输入词元之间的多体相互作用,低阶相互作用的预测误差会先达到饱和点,而高阶相互作用的学习仍在持续。为开展这项分析,我们提出了一种生成给定自然语言数据集“克隆体”的流程,能够严格捕捉词元之间最高到指定阶数的相互作用。该方法为研究自然语言处理及其他领域中,数据里不同阶数的相互作用如何影响学习过程提供了新的可能。
英文摘要:
The remarkable capability of over-parameterised neural networks to generalise effectively has been explained by invoking a ``simplicity bias'': neural networks prevent overfitting by initially learning simple classifiers before progressing to more complex, non-linear functions. While simplicity biases have been described theoretically and experimentally in feed-forward networks for supervised learning, the extent to which they also explain the remarkable success of transformers trained with self-supervised techniques remains unclear. In our study, we demonstrate that transformers, trained on natural language data, also display a simplicity bias. Specifically, they sequentially learn many-body interactions among input tokens, reaching a saturation point in the prediction error for low-degree interactions while continuing to learn high-degree interactions. To conduct this analysis, we develop a procedure to generate \textit{clones} of a given natural language data set, which rigorously capture the interactions between tokens up to a specified order. This approach opens up the possibilities of studying how interactions of different orders in the data affect learning, in natural language processing and beyond.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。