In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.
Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer
Korea University(高丽大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Sogang University(西江大学);GIST(光州科学技术院);Yonsei University(延世大学);Carnegie Mellon University(卡内基梅隆大学)
ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws
NII LLMC;Osaka University(大阪大学);Nara Institute of Science and Technology(奈良科学技術大學);University College London(伦敦大学学院);Kyoto University(京都大学);Waseda University(早稻田大学)
Finding Concept-specific Biases in Form--Meaning Associations
University of Cambridge(剑桥大学);Google(谷歌);Kazan Federal University(喀山联邦大学);ETH Zürich(苏黎世联邦理工学院);Harvard University(哈佛大学);MPI for Evolutionary Anthropology(马克斯·普朗克进化人类学研究所);HSE University(高等经济大学)
It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
University of Copenhagen(哥本哈根大学);Johns Hopkins University(约翰斯·霍普金斯大学);Carnegie Mellon University(卡内基梅隆大学);University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院);Tokyo Institute of Technology(东京工业大学)
An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models
Indian Institute of Technology Madras(印度马德拉斯理工学院);National Institute of Information and Communications Technology, Japan(日本信息通信研究机构);Indian Institute of Technology Bombay(印度孟买理工学院);IT University of Copenhagen, Denmark(丹麦哥本哈根信息技术大学);Microsoft India(微软印度公司);Nilekani Centre at AI4Bharat(AI4Bharat的尼莱卡尼中心)
本文针对多语言语言模型分词器词汇扩展中的嵌入初始化问题,提出无需跨语言嵌入的约束 Word2Vec(CW2V)方法,并在 RoBERTa 和 LLaMA 2 上验证其与先进方法相当甚至更优。
Toward Robust Multilingual Adaptation of LLMs for Low-Resource Languages
Department of Automation, Tsinghua University, Beijing, China(清华大学自动化系);Alibaba International Digital Commerce Group, Beijing, China(阿里巴巴国际数字 commerce 集团);School of Software, Tsinghua University, Beijing, China(清华大学软件学院)
Minimax and Neyman-Pearson Meta-Learning for Outlier Languages
Mila – Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);University of Copenhagen(哥本哈根大学);University of Montreal(蒙特利尔大学)