arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.07834cs.CL
收藏

InfoXLM:一个跨语言语言模型预训练的信息论框架

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Microsoft Corporation(微软公司);Beijing Institute of Technology(北京理工大学)

Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou

摘要:

在这项工作中,我们提出了一个信息理论的框架,制定跨语言的语言模型预训练最大限度地提高多语言多粒度文本之间的互信息。统一的观点有助于我们更好地理解现有的跨语言表征学习方法。更重要的是,受该框架的启发,我们提出了一种新的基于对比学习的预训练任务。具体来说,我们认为双语句子对作为两个相同的意义的观点,并鼓励他们的编码表示比否定的例子更相似。通过利用单语和平行语料库,我们联合训练借口任务,以提高预训练模型的跨语言可移植性。几个基准测试的实验结果表明,我们的方法取得了相当好的性能。代码和预训练模型可在https://aka.ms/infoxlm上获得。

英文摘要:

In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑