利用知识蒸馏压缩多语言神经机器翻译模型的实证研究
An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models
AI总结：
本文实证研究用知识蒸馏压缩多语言神经机器翻译模型，以印度语到英语翻译为例，分析模型缩小4-5倍时最高3.5 BLEU的性能损失，并评估深度、参数共享、多阶段训练和适配器等改进设计。
摘要：
知识蒸馏（KD）是一种众所周知的压缩神经模型的方法。然而，尽管多语言神经机器翻译（MNMT）广受欢迎且具有优越性，但专注于将知识从大型MNMT模型蒸馏到较小模型的研究实际上并不存在。本文通过对用于压缩MNMT模型的知识蒸馏进行实证研究来弥合这一差距。我们以印度语到英语的翻译为案例研究，证明常用的语言无关和语言感知KD方法所产生的模型体积缩小了4-5倍，但也会出现高达3.5 BLEU的性能下降。为缓解这一问题，我们随后对多种设计考量进行了实验，例如较浅模型与较深模型、重度参数共享、多阶段训练以及适配器。我们观察到，更深的紧凑模型往往与较浅的非紧凑模型一样好，并且在高质量子集上微调蒸馏模型会略微提升翻译质量。总体而言，我们得出结论：通过KD压缩MNMT模型具有挑战性，这表明未来研究仍有巨大空间。
英文摘要：
Knowledge distillation (KD) is a well-known method for compressing neural models. However, works focusing on distilling knowledge from large multilingual neural machine translation (MNMT) models into smaller ones are practically nonexistent, despite the popularity and superiority of MNMT. This paper bridges this gap by presenting an empirical investigation of knowledge distillation for compressing MNMT models. We take Indic to English translation as a case study and demonstrate that commonly used language-agnostic and language-aware KD approaches yield models that are 4-5x smaller but also suffer from performance drops of up to 3.5 BLEU. To mitigate this, we then experiment with design considerations such as shallower versus deeper models, heavy parameter sharing, multi-stage training, and adapters. We observe that deeper compact models tend to be as good as shallower non-compact ones, and that fine-tuning a distilled model on a High-Quality subset slightly boosts translation quality. Overall, we conclude that compressing MNMT models via KD is challenging, indicating immense scope for further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
