arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1912.03334cs.CL
收藏

神经机器翻译中序列级知识蒸馏的数据增强

Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation

Johns Hopkins University(约翰斯·霍普金斯大学)

Mitchell A. Gordon, Kevin Duh

摘要:

序列级知识蒸馏(SLKD)是一种模型压缩技术,它利用大型、准确的教师模型来训练较小的、参数化不足的学生模型。为什么使用SLKD预处理MT数据有助于我们训练更小的模型?我们测试的共同假设,SLKD解决能力不足的学生通过“简化”嘈杂的数据点,并发现它不太可能在我们的情况下。在原始数据集和“简化”数据集的连接上训练的模型与基线SLKD一样通用。然后,我们提出了一个替代假设的镜头下的数据增强和正则化。我们尝试了各种增强策略,并观察到dropout正则化可能变得不必要。我们的方法在TED演讲中实现了0.7-1.2的BLEU增益。

英文摘要:

Sequence-level knowledge distillation (SLKD) is a model compression technique that leverages large, accurate teacher models to train smaller, under-parameterized student models. Why does pre-processing MT data with SLKD help us train smaller models? We test the common hypothesis that SLKD addresses a capacity deficiency in students by "simplifying" noisy data points and find it unlikely in our case. Models trained on concatenations of original and "simplified" datasets generalize just as well as baseline SLKD. We then propose an alternative hypothesis under the lens of data augmentation and regularization. We try various augmentation strategies and observe that dropout regularization can become unnecessary. Our methods achieve BLEU gains of 0.7-1.2 on TED Talks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑