M2G-Eval: 提升和评估多粒度多语言代码生成
M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation
AI总结：
M2G-Eval通过多粒度多语言框架评估代码生成能力，揭示了不同粒度任务的难度差异和跨语言学习的关联性。
摘要：
代码大语言模型（LLMs）的快速发展引发了对系统评估其代码生成能力的广泛关注，但现有基准测试大多仅在单一结构粒度上评估模型，并且专注于有限的编程语言，这掩盖了不同代码范围和多语言场景下的细粒度能力差异。我们引入M2G-Eval，一种多粒度、多语言框架，用于评估大型语言模型（LLMs）在四个层面的代码生成能力：类、函数、块和行。涵盖18种编程语言，M2G-Eval包含17000多个训练任务和1286个人注释、受控污染的测试实例。我们通过监督微调和群体相对策略优化训练Qwen3-8B，开发出M2G-Eval-Coder模型。评估30个模型（28个最先进的LLM加上我们的两个M2G-Eval-Coder变体）揭示了三个主要发现：（1）存在明显的难度层次，行级任务最简单，类级任务最困难；（2）随着任务复杂度增加，全粒度和部分粒度语言之间的性能差距逐渐扩大；（3）强跨语言相关性，表明模型学习了可转移的编程概念。M2G-Eval能够对代码生成能力进行细粒度诊断，并突显了在合成复杂、长形式代码方面的持续挑战。
英文摘要：
The rapid advancement of code large language models (LLMs) has sparked significant research interest in systematically evaluating their code generation capabilities, yet existing benchmarks predominantly assess models at a single structural granularity and focus on limited programming languages, obscuring fine-grained capability variations across different code scopes and multilingual scenarios. We introduce M2G-Eval, a multi-granularity, multilingual framework for evaluating code generation in large language models (LLMs) across four levels: Class, Function, Block, and Line. Spanning 18 programming languages, M2G-Eval includes 17K+ training tasks and 1,286 human-annotated, contamination-controlled test instances. We develop M2G-Eval-Coder models by training Qwen3-8B with supervised fine-tuning and Group Relative Policy Optimization. Evaluating 30 models (28 state-of-the-art LLMs plus our two M2G-Eval-Coder variants) reveals three main findings: (1) an apparent difficulty hierarchy, with Line-level tasks easiest and Class-level most challenging; (2) widening performance gaps between full- and partial-granularity languages as task complexity increases; and (3) strong cross-language correlations, suggesting that models learn transferable programming concepts. M2G-Eval enables fine-grained diagnosis of code generation capabilities and highlights persistent challenges in synthesizing complex, long-form code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
