面向非自回归神经机器翻译的选择性知识蒸馏
Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation
AI总结：
本文提出一种选择性知识蒸馏方法，通过引入NAT评估器选择高质量且易学的目标，并结合渐进式蒸馏，有效缓解教师错误传播问题，在多个NAT模型上实现翻译质量与数据复杂度的灵活权衡。
摘要：
得益于序列级知识蒸馏，非自回归 Transformer (NAT) 在神经机器翻译任务中取得了巨大成功。然而，现有的知识蒸馏存在副作用，例如将教师模型的错误传播给 NAT 学生模型，这可能限制 NAT 模型的进一步提升，且在现有研究中鲜少被讨论。在本文中，我们通过引入 NAT 评估器来选择高质量且易于学习的 NAT 友好目标，从而提出选择性知识蒸馏。此外，我们引入了一种简单而有效的渐进式蒸馏方法以提升 NAT 性能。在多个 WMT 语言方向和几种代表性 NAT 模型上的实验结果表明，我们的方法能够在 NAT 模型训练数据的质量和复杂度之间实现灵活的权衡，取得强劲的性能表现。进一步分析表明，仅蒸馏 5% 的原始翻译即可帮助 NAT 模型在 BLEU 指标上比在原始数据上训练的对应模型高出约 2.4。
英文摘要：
Benefiting from the sequence-level knowledge distillation, the Non-Autoregressive Transformer (NAT) achieves great success in neural machine translation tasks. However, existing knowledge distillation has side effects, such as propagating errors from the teacher to NAT students, which may limit further improvements of NAT models and are rarely discussed in existing research. In this paper, we introduce selective knowledge distillation by introducing an NAT evaluator to select NAT-friendly targets that are of high quality and easy to learn. In addition, we introduce a simple yet effective progressive distillation method to boost NAT performance. Experiment results on multiple WMT language directions and several representative NAT models show that our approach can realize a flexible trade-off between the quality and complexity of training data for NAT models, achieving strong performances. Further analysis shows that distilling only 5% of the raw translations can help an NAT outperform its counterpart trained on raw data by about 2.4 BLEU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
