基于多语言编码器与知识蒸馏的低资源神经机器翻译增强：一项案例研究
Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study
AI总结：
针对mBART-50等预训练多语言模型覆盖语种有限、扩展易出现灾难性遗忘的问题，提出结合多语言编码器seq2seq架构与互补知识蒸馏的低资源NMT框架，在印度低资源语言翻译任务上指标显著提升，代码已开源。
摘要：
神经机器翻译（NMT）仍是一项艰巨挑战，尤其是在处理低资源语言时。预训练的序列到序列（seq2seq）多语言模型（如mBART-50）已在各类低资源NMT任务中展现出优异性能。但这类模型的预训练仅覆盖50种语言，无法支持众多低资源语言，尤其是印度次大陆的语种。扩展mBART-50的语言支持需复杂预训练，还可能因灾难性遗忘导致性能下降。针对这些扩展难题，本文探索了一个在seq2seq架构中结合预训练语言模型优势与知识蒸馏的框架，以助力低资源语言（包括mBART-50未覆盖的语种）的翻译。该框架以基于多语言编码器的seq2seq模型为基础架构，随后采用互补知识蒸馏技术缓解训练不均衡的影响。我们在三种低资源印度语言的四个印地语互译方向上对框架进行评估，相比基线模型在BLEU-4和chrF指标上取得显著提升。此外，我们开展了人工评估以验证方法的有效性。我们的代码已公开于https://github.com/raypretam/Two-step-low-res-NMT。
英文摘要：
Neural Machine Translation (NMT) remains a formidable challenge, especially when dealing with low-resource languages. Pre-trained sequence-to-sequence (seq2seq) multi-lingual models, such as mBART-50, have demonstrated impressive performance in various low-resource NMT tasks. However, their pre-training has been confined to 50 languages, leaving out support for numerous low-resource languages, particularly those spoken in the Indian subcontinent. Expanding mBART-50's language support requires complex pre-training, risking performance decline due to catastrophic forgetting. Considering these expanding challenges, this paper explores a framework that leverages the benefits of a pre-trained language model along with knowledge distillation in a seq2seq architecture to facilitate translation for low-resource languages, including those not covered by mBART-50. The proposed framework employs a multilingual encoder-based seq2seq model as the foundational architecture and subsequently uses complementary knowledge distillation techniques to mitigate the impact of imbalanced training. Our framework is evaluated on three low-resource Indic languages in four Indic-to-Indic directions, yielding significant BLEU-4 and chrF improvements over baselines. Further, we conduct human evaluation to confirm effectiveness of our approach. Our code is publicly available at https://github.com/raypretam/Two-step-low-res-NMT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
