arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.13678cs.CL
收藏

神经机器翻译领域自适应的剪枝扩展模型

Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation

University of Chinese Academy of Sciences(中国科学院大学);Beijing Language and Culture University(北京语言大学)

Shuhao Gu, Yang Feng, Wanying Xie

摘要:

领域自适应算法广泛应用于神经机器翻译的实际应用中,其目标是在一般域和域内都获得良好的翻译性能。然而,现有的领域自适应方法通常遭受灾难性遗忘、领域发散和模型爆炸。为了解决这三个问题,我们提出了一种基于神经元或参数在翻译模型中的重要性的“分而治之”的方法。在我们的方法中,我们首先修剪模型,只保留重要的神经元或参数,使它们负责一般域和域内翻译。然后利用知识蒸馏方法,在原模型的监督下,进一步训练剪枝后的模型。最后,我们将模型扩展到原始大小,并微调域内转换的添加参数。我们在不同的语言和领域上进行了实验,结果表明,与几个强基线相比,我们的方法可以实现显着的改进。

英文摘要:

Domain Adaptation is widely used in practical applications of neural machine translation, which aims to achieve good performance on both the general-domain and in-domain. However, the existing methods for domain adaptation usually suffer from catastrophic forgetting, domain divergence, and model explosion. To address these three problems, we propose a method of "divide and conquer" which is based on the importance of neurons or parameters in the translation model. In our method, we first prune the model and only keep the important neurons or parameters, making them responsible for both general-domain and in-domain translation. Then we further train the pruned model supervised by the original unpruned model with the knowledge distillation method. Last we expand the model to the original size and fine-tune the added parameters for the in-domain translation. We conduct experiments on different languages and domains and the results show that our method can achieve significant improvements compared with several strong baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑