arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.07276cs.CL
收藏

序列长度是一个域:Transformer模型中基于长度的过拟合

Sequence Length is a Domain: Length-based Overfitting in Transformer Models

Charles University(查理大学)

Dušan Variš, Ondřej Bojar

摘要:

基于transformer的序列到序列架构虽然在大量NLP任务上实现了最先进的结果,但在训练过程中仍然会受到过拟合的影响。在实践中,这通常通过应用正则化方法(例如dropout,L2正则化)或通过提供大量的训练数据来解决。此外,已知Transformer和其他架构在生成非常长的序列时会遇到困难。例如,在机器翻译中,与之前基于短语的翻译方法相比,基于神经的系统在非常长的序列上表现更差(Koehn和Knowles,2017)。我们目前的结果表明,这个问题也可能是在训练和验证数据的长度分布之间的不匹配,结合上述倾向的神经网络过拟合的训练数据。我们证明了一个简单的字符串编辑任务和机器翻译任务的Transformer模型的性能显着下降时,面对序列的长度偏离的长度分布在训练数据。此外,我们发现,观察到的性能下降是由于假设长度对应于模型在训练过程中看到的长度,而不是输入序列的长度。

英文摘要:

Transformer-based sequence-to-sequence architectures, while achieving state-of-the-art results on a large number of NLP tasks, can still suffer from overfitting during training. In practice, this is usually countered either by applying regularization methods (e.g. dropout, L2-regularization) or by providing huge amounts of training data. Additionally, Transformer and other architectures are known to struggle when generating very long sequences. For example, in machine translation, the neural-based systems perform worse on very long sequences when compared to the preceding phrase-based translation approaches (Koehn and Knowles, 2017). We present results which suggest that the issue might also be in the mismatch between the length distributions of the training and validation data combined with the aforementioned tendency of the neural networks to overfit to the training data. We demonstrate on a simple string editing task and a machine translation task that the Transformer model performance drops significantly when facing sequences of length diverging from the length distribution in the training data. Additionally, we show that the observed drop in performance is due to the hypothesis length corresponding to the lengths seen by the model during training rather than the length of the input sequence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑