M2rc-Eval：大规模多语言仓库级代码补全评估
M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
AI总结：
本文提出覆盖18种编程语言、提供AST驱动细粒度场景标注的 M2RC-EVAL 基准，并构建 M2RC-INSTRUCT 指令数据集，以评估并提升代码LLM的多语言仓库级补全能力。
摘要：
仓库级代码补全已在软件工程领域引起广泛关注，并已有若干基准数据集被提出。然而，现有仓库级代码补全基准通常只关注数量有限的编程语言（少于5种），无法评估现有代码大型语言模型（LLM）在不同语言之间的通用代码智能能力。此外，现有基准通常报告不同语言的总体平均分，而忽略了不同补全场景中的细粒度能力。因此，为促进代码LLM在多语言场景下的研究，我们提出了一个覆盖18种编程语言的大规模多语言仓库级代码补全基准（称为 M2RC-EVAL），并针对不同补全场景提供两类细粒度标注（即桶级标注和语义级标注），这些标注是基于解析得到的抽象语法树获得的。此外，我们还构建了一个大规模多语言指令语料库 M2RC-INSTRUCT 数据集，以提升现有代码LLM的仓库级代码补全能力。综合实验结果证明了 M2RC-EVAL 和 M2RC-INSTRUCT 的有效性。
英文摘要：
Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced. However, existing repository-level code completion benchmarks usually focus on a limited number of languages (<5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs). Besides, the existing benchmarks usually report overall average scores of different languages, where the fine-grained abilities in different completion scenarios are ignored. Therefore, to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M2RC-EVAL), and two types of fine-grained annotations (i.e., bucket-level and semantic-level) on different completion scenarios are provided, where we obtain these annotations based on the parsed abstract syntax tree. Moreover, we also curate a massively multilingual instruction corpora M2RC- INSTRUCT dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
