BERT、mBERT或BiBERT？神经机器翻译的语境化嵌入研究
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
摘要：
双向编码器在许多自然语言处理任务中使用掩蔽语言模型（例如BERT）的成功促使研究人员尝试将这些预先训练的模型纳入神经机器翻译（NMT）系统。然而，所提出的用于合并预训练模型的方法是不平凡的，并且主要集中在BERT上，其缺乏对其他预训练模型可能对翻译性能的影响的比较。在本文中，我们证明了简单地使用定制和合适的双语预训练语言模型（称为BiBERT）的输出（上下文嵌入）作为NMT编码器的输入，可以实现最先进的翻译性能。此外，我们还提出了一个随机层选择方法和双向翻译模型的概念，以确保充分利用上下文嵌入。在不使用反向翻译的情况下，我们的最佳模型在IWITOS '14数据集上实现了30.45的BLEU分数，对于En-> De和38.61的De->En，以及WMT'14数据集上的31.26的En->De和34.94的De->En，这超过了所有公布的数字。
英文摘要：
The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incorporate these pre-trained models into neural machine translation (NMT) systems. However, proposed methods for incorporating pre-trained models are non-trivial and mainly focus on BERT, which lacks a comparison of the impact that other pre-trained models may have on translation performance. In this paper, we demonstrate that simply using the output (contextualized embeddings) of a tailored and suitable bilingual pre-trained language model (dubbed BiBERT) as the input of the NMT encoder achieves state-of-the-art translation performance. Moreover, we also propose a stochastic layer selection approach and a concept of dual-directional translation model to ensure the sufficient utilization of contextualized embeddings. In the case of without using back translation, our best models achieve BLEU scores of 30.45 for En->De and 38.61 for De->En on the IWSLT'14 dataset, and 31.26 for En->De and 34.94 for De->En on the WMT'14 dataset, which exceeds all published numbers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
