arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.14928cs.CL
收藏

低资源神经机器翻译的语言模型先验

Language Model Prior for Low-Resource Neural Machine Translation

Institute for Language, Cognition and Computation(语言、认知与计算研究所);School of Informatics, University of Edinburgh(爱丁堡大学信息学院)

Christos Baziotis, Barry Haddow, Alexandra Birch

摘要:

大型并行语料库的缺乏是神经机器翻译的一个重要障碍。一种常见的解决方案是利用在丰富的单语数据上训练的语言模型(LM)的知识。在这项工作中,我们提出了一种新的方法,将LM作为神经翻译模型(TM)的先验。具体来说,我们添加了一个正则化项,它将TM的输出分布推到LM先验下的可能性,同时避免TM与LM“不一致”时的错误预测。这一目标涉及到知识蒸馏,其中LM可以被视为教导TM关于目标语言。所提出的方法不会影响解码速度,因为LM仅在训练时使用,不像以前的工作需要在推理过程中使用。我们提出了一个分析的影响,不同的方法对TM的分布。两个低资源机器翻译数据集的结果显示出明显的改善,即使是有限的单语数据。

英文摘要:

The scarcity of large parallel corpora is an important obstacle for neural machine translation. A common solution is to exploit the knowledge of language models (LM) trained on abundant monolingual data. In this work, we propose a novel approach to incorporate a LM as prior in a neural translation model (TM). Specifically, we add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior, while avoiding wrong predictions when the TM "disagrees" with the LM. This objective relates to knowledge distillation, where the LM can be viewed as teaching the TM about the target language. The proposed approach does not compromise decoding speed, because the LM is used only at training time, unlike previous work that requires it during inference. We present an analysis of the effects that different methods have on the distributions of the TM. Results on two low-resource machine translation datasets show clear improvements even with limited monolingual data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑