ERNIE-Code：超越以英语为中心的编程语言跨语言预训练
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages
AI总结：
发布支持116种自然语言和6种编程语言的统一预训练模型ERNIE-Code，采用span-corruption和基于枢轴的翻译语言建模方法，在代码智能任务及零样本提示中表现优越。
摘要：
使用相同编程语言（PL）的软件工程师可能说不同的自然语言（NL），反之亦然，这给沟通和工作效率造成了巨大障碍。最近的研究证明了生成式预训练在计算机程序中的有效性，但它们始终以英语为中心。在这项工作中，我们致力于为大语言模型（LLM）弥合多语言NL与多语言PL之间的鸿沟。我们发布了ERNIE-Code，一个支持116种NL和6种PL的统一预训练语言模型。我们采用两种方法进行通用跨语言预训练：从单语言NL或PL中学习模式的span-corruption语言建模；以及依赖多种NL和PL平行数据的基于枢轴的翻译语言建模。广泛的结果表明，ERNIE-Code在代码智能的各种端到端任务中超越了以往针对PL或NL的多语言LLM，包括多语言代码到文本、文本到代码、代码到代码和文本到文本生成。我们进一步展示了其在多语言代码摘要和文本到文本翻译上的零样本提示优势。我们发布了代码和预训练检查点。
英文摘要：
Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
