We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
Microsoft(微软公司)
在本文中,我们介绍了XGLUE,这是一个新的基准数据集,可用于使用多语言和双语语料库训练大规模跨语言预训练模型,并评估它们在各种跨语言任务中的性能。与GLUE(Wang等人,2019),仅针对自然语言理解任务进行英文标注,XGLUE主要有两大优势:(1)提供11个多样化任务,涵盖自然语言理解和生成场景;(2)针对每个任务,提供多种语言的标注数据。我们扩展了最近的跨语言预训练模型Unicoder(Huang et al.,2019年),以涵盖理解和生成任务,这是对XGLUE作为一个强大的基线进行评估。我们还评估了多语言BERT,XLM和XLM-R的基础版本(12层)进行比较。
KnowCoder-X: Boosting Multilingual Information Extraction via Code
Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences(网络数据科学与技术重点实验室,计算技术研究所,中国科学院);State Key Laboratory of AI Safety(人工智能安全国家重点实验室);School of Computer Science, University of Chinese Academy of Sciences(中国科学院大学计算机科学学院);School of Software, Northeastern University(东北大学软件学院);School of Software, Peking University(北京大学软件学院)
Cross-lingual Similarity of Multilingual Representations Revisited
University of Tartu(塔尔图大学);Institute of Computer Science(计算机科学研究所)
相关工作使用CKA和CCA的变体等索引来度量多语言语言模型中跨语言表示的相似性。在本文中,我们认为CKA/CCA的假设与跨语言学习分析的激励目标之一,即,解释zero-shot跨语言迁移。我们强调什么跨语言的相似性,这些指标未能捕捉和提供一个激励性的案例研究\textit{demonstrating the problem empirically}有价值的方面。然后,我们介绍了\textit{Average Neuron-Wise Correlation (ANC)}作为一个简单的替代方案,是免除CKA/CCA的困难,是很好的,特别是在跨语言的背景下。最后,我们使用ANC来构造证据,证明先前引入的“先对齐,然后预测”模式不仅发生在掩蔽语言模型(MLMs)中,而且发生在具有\textit{causal language modeling}目标的多语言模型(CLMs)中。此外,我们表明,该模式扩展到\textit{scaled versions}的MLMs和CLMs(高达85倍的原始mBERT)。\footnote{Our code is publicly available at \url{https://github.com/TartuNLP/xsim}}
MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Stability AI(Stability AI公司);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));TU Darmstadt(达姆施塔特工业大学);Aleph Alpha(Aleph Alpha公司);University of Texas(得克萨斯大学);LAION(大规模人工智能开放网络(LAION))