arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2010.03017cs.CLcs.LG
收藏

多语言模型中的负干扰:发现和元学习处理

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Carnegie Mellon University(卡内基梅隆大学)

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

摘要:

现代多语言模型是在来自多种语言的连接文本上训练的,希望能为每种语言带来好处(正迁移),其中最明显的好处是低资源语言。然而,最近的研究表明,这种方法会降低高资源语言的性能,这种现象称为负干扰。在本文中,我们提出了第一个系统的研究负干扰。我们发现,与以前的信念相反,负面干扰也会影响低资源语言。虽然最大限度地共享参数来学习语言通用结构,但我们证明了语言特定的参数确实存在于多语言模型中,并且它们是负面干扰的潜在原因。出于这些观察,我们还提出了一个Meta学习算法,获得更好的跨语言的可移植性和消除负面干扰,通过添加语言特定的层作为元参数和训练他们的方式,明确提高共享层的泛化所有语言。总的来说,我们的研究结果表明,负面干扰比以前已知的更常见,这为改善多语言表示提出了新的方向。

英文摘要:

Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages. However, recent work has shown that this approach can degrade performance on high-resource languages, a phenomenon known as negative interference. In this paper, we present the first systematic study of negative interference. We show that, contrary to previous belief, negative interference also impacts low-resource languages. While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific parameters do exist in multilingual models and they are a potential cause of negative interference. Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference, by adding language-specific layers as meta-parameters and training them in a manner that explicitly improves shared layers' generalization on all languages. Overall, our results show that negative interference is more common than previously known, suggesting new directions for improving multilingual representations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑