arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.01994cs.LGcs.CL
收藏

NMT中的数据缩放定律:噪声和结构的影响

Data Scaling Laws in NMT: The Effect of Noise and Architecture

Harvard University(哈佛大学);Google(谷歌);University of Edinburgh(爱丁堡大学)

Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, Orhan Firat

摘要:

在这项工作中,我们研究了不同的架构和训练数据质量对神经机器翻译(NMT)的数据缩放特性的影响。首先,我们建立了编码器-解码器Transformer模型的测试损失在训练样本的数量上按幂律缩放,并依赖于模型的大小。然后,我们系统地改变训练设置的各个方面,以了解它们如何影响数据缩放定律。特别是,我们改变了以下内容:(1)架构和任务设置:我们与transformer-LSTM混合,以及具有语言建模损失的仅解码器Transformer进行比较(2)训练分布中的噪声水平:我们实验过滤,并添加iid合成噪声。在上述所有情况下,我们发现数据缩放指数受到的影响最小,这表明稍微差一点的架构或训练数据可以通过添加更多数据来补偿。最后,我们发现,使用反向转换的数据,而不是并行数据,可以显着降低标度指数。

英文摘要:

In this work, we study the effect of varying the architecture and training data quality on the data scaling properties of Neural Machine Translation (NMT). First, we establish that the test loss of encoder-decoder transformer models scales as a power law in the number of training samples, with a dependence on the model size. Then, we systematically vary aspects of the training setup to understand how they impact the data scaling laws. In particular, we change the following (1) Architecture and task setup: We compare to a transformer-LSTM hybrid, and a decoder-only transformer with a language modeling loss (2) Noise level in the training distribution: We experiment with filtering, and adding iid synthetic noise. In all the above cases, we find that the data scaling exponents are minimally impacted, suggesting that marginally worse architectures or training data can be compensated for by adding more data. Lastly, we find that using back-translated data instead of parallel data, can significantly degrade the scaling exponent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑