arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.13619cs.CL
收藏

基于参数微分的多语言神经机器翻译

Parameter Differentiation based Multilingual Neural Machine Translation

National Laboratory of Pattern Recognition, CASIA(模式识别国家重点实验室,中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)

Qian Wang, Jiajun Zhang

摘要:

多语言神经机器翻译(MNMT)旨在用单个模型翻译多种语言,并且由于不同语言之间具有共享参数的有效知识转移而被证明是成功的。然而,哪些参数应该共享,哪些参数需要针对特定任务,这仍然是一个悬而未决的问题。目前,普遍的做法是对特定语言的模块进行抽象设计或搜索,很难找到最佳配置。在本文中,我们提出了一种新的基于参数微分的方法,该方法允许模型在训练过程中确定哪些参数应该是语言特定的。受细胞分化的启发,我们方法中的每个共享参数都可以动态地分化为更专业的类型。我们进一步定义的区分标准为任务间的梯度相似性。因此,具有冲突的任务间梯度的参数更可能是语言特定的。在多语言数据集上的大量实验表明,我们的方法显着优于具有不同参数共享配置的各种强基线。进一步的分析表明,由我们的方法得到的参数共享配置与语言的邻近性相关。

英文摘要:

Multilingual neural machine translation (MNMT) aims to translate multiple languages with a single model and has been proved successful thanks to effective knowledge transfer among different languages with shared parameters. However, it is still an open question which parameters should be shared and which ones need to be task-specific. Currently, the common practice is to heuristically design or search language-specific modules, which is difficult to find the optimal configuration. In this paper, we propose a novel parameter differentiation based method that allows the model to determine which parameters should be language-specific during training. Inspired by cellular differentiation, each shared parameter in our method can dynamically differentiate into more specialized types. We further define the differentiation criterion as inter-task gradient similarity. Therefore, parameters with conflicting inter-task gradients are more likely to be language-specific. Extensive experiments on multilingual datasets have demonstrated that our method significantly outperforms various strong baselines with different parameter sharing configurations. Further analyses reveal that the parameter sharing configuration obtained by our method correlates well with the linguistic proximities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑