arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.00411cs.CL
收藏

质量评价的知识蒸馏

Knowledge Distillation for Quality Estimation

University of Sheffield(谢菲尔德大学);Imperial College London(帝国理工学院);University of Wolverhampton(伍尔弗汉普顿大学)

Amit Gajbhiye, Marina Fomicheva, Fernando Alva-Manchego, Frédéric Blain, Abiola Obamuyide, Nikolaos Aletras, Lucia Specia

摘要:

质量评估(QE)是在没有参考翻译的情况下自动预测机器翻译质量的任务,使其适用于实时设置,例如翻译在线社交媒体对话。QE最近的成功源于使用多语言预训练表示,其中非常大的模型导致了令人印象深刻的结果。然而,这种模型的推理时间,磁盘和内存要求不允许在现实世界中广泛使用。对于许多使用场景来说,在提取的预训练表示上训练的模型仍然非常大。相反,我们建议直接将知识从强大的QE教师模型转移到具有不同的更浅架构的更小的模型。我们表明,这种方法与数据增强相结合,可以产生轻量级的QE模型,这些模型与参数少8倍的蒸馏预训练表示相比具有竞争力。

英文摘要:

Quality Estimation (QE) is the task of automatically predicting Machine Translation quality in the absence of reference translations, making it applicable in real-time settings, such as translating online social media conversations. Recent success in QE stems from the use of multilingual pre-trained representations, where very large models lead to impressive results. However, the inference time, disk and memory requirements of such models do not allow for wide usage in the real world. Models trained on distilled pre-trained representations remain prohibitively large for many usage scenarios. We instead propose to directly transfer knowledge from a strong QE teacher model to a much smaller model with a different, shallower architecture. We show that this approach, in combination with data augmentation, leads to light-weight QE models that perform competitively with distilled pre-trained representations with 8x fewer parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑