面向低资源语言的多语言神经翻译模型多假设蒸馏
Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
AI总结：
针对多语言翻译模型序列级知识蒸馏仅依赖束搜索单一结果的局限，提出多假设蒸馏方法，利用多译文提升学生模型性能并缓解低资源语言场景下的性别偏差放大问题。
摘要：
本文探究了多语言预训练编码器-解码器翻译模型的序列级知识蒸馏（KD，Knowledge Distillation）。我们认为，教师模型的输出分布能为学生模型提供束搜索（标准解码方法）得到的近似众数之外的宝贵信息，据此提出多假设蒸馏（MHD，Multi-Hypothesis Distillation）——一种为每个源句生成多个译文的序列级KD方法。该方法能更全面地呈现教师模型的分布，让学生模型接触到更广泛的目标端前缀。我们利用束搜索生成的n-best列表引导学生模型学习，并研究了替代解码方法以解决变异性低、低频词元表征不足等问题。针对低资源语言的研究表明，尽管采样方法相较于基于束搜索的方法可能会略微降低翻译质量，但它们能提升生成语料的变异性和词汇丰富度，最终提升学生模型性能，并缓解知识蒸馏常伴随的性别偏差放大问题。
英文摘要：
This paper explores sequence-level knowledge distillation (KD) of multilingual pre-trained encoder-decoder translation models. We argue that the teacher model's output distribution holds valuable insights for the student, beyond the approximated mode obtained through beam search (the standard decoding method), and present Multi-Hypothesis Distillation (MHD), a sequence-level KD method that generates multiple translations for each source sentence. This provides a larger representation of the teacher model distribution and exposes the student model to a wider range of target-side prefixes. We leverage $n$-best lists from beam search to guide the student's learning and examine alternative decoding methods to address issues like low variability and the under-representation of infrequent tokens. For low-resource languages, our research shows that while sampling methods may slightly compromise translation quality compared to beam search based approaches, they enhance the generated corpora with greater variability and lexical richness. This ultimately improves student model performance and mitigates the gender bias amplification often associated with KD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
