arXiv 2411.023102025-02-24cs.CL

MdEval：大规模多语言代码调试

MdEval: Massively Multilingual Code Debugging

CCSE, Beihang University（北京航空航天大学 CCSE）

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liq… 展开作者

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, Zhoujun Li

备注：: 15 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出首个大规模多语言调试基准MdEval，含18种语言3.6K样本，覆盖APR、CR、BI任务，并构建指令语料库训练xDebugCoder，实验显示开源与闭源模型性能差距大。

摘要：

代码大语言模型（LLMs）通过直接根据错误代码片段生成正确代码，在代码调试方面取得了显著进展。编程基准测试通常由错误代码片段及其相关测试用例组成，用于评估LLMs的调试能力。然而，许多现有基准主要关注Python，且在语言多样性方面往往受限（例如DebugBench和DebugEval）。为了推动LLMs在多语言调试领域的发展，我们提出了首个大规模多语言调试基准，该基准包含18种编程语言的3.6K测试样本，并覆盖了自动程序修复（APR）任务、代码审查（CR）任务和缺陷识别（BI）任务。此外，我们通过将缺陷注入正确的多语言查询和解决方案中（xDebugGen），引入了调试指令语料库MDEVAL-INSTRUCT。进一步地，我们训练了一个多语言调试器xDebugCoder，该调试器基于MDEVAL-INSTRUCT作为强基线，专门处理多种编程语言的缺陷（例如Rust语言中的“Missing Mut”和C语言中的“Misused Macro Definition”）。我们在MDEVAL上的大量实验揭示了开源模型与闭源LLMs（例如GPT和Claude系列）之间的显著性能差距，凸显了多语言代码调试场景中巨大的改进空间。

英文摘要：

Code large language models (LLMs) have made significant progress in code debugging by directly generating the correct code based on the buggy code snippet. Programming benchmarks, typically consisting of buggy code snippet and their associated test cases, are used to assess the debugging capabilities of LLMs. However, many existing benchmarks primarily focus on Python and are often limited in terms of language diversity (e.g., DebugBench and DebugEval). To advance the field of multilingual debugging with LLMs, we propose the first massively multilingual debugging benchmark, which includes 3.6K test samples of 18 programming languages and covers the automated program repair (APR) task, the code review (CR) task, and the bug identification (BI) task. Further, we introduce the debugging instruction corpora MDEVAL-INSTRUCT by injecting bugs into the correct multilingual queries and solutions (xDebugGen). Further, a multilingual debugger xDebugCoder trained on MDEVAL-INSTRUCT as a strong baseline specifically to handle the bugs of a wide range of programming languages (e.g. "Missing Mut" in language Rust and "Misused Macro Definition" in language C). Our extensive experiments on MDEVAL reveal a notable performance gap between open-source models and closed-source LLMs (e.g., GPT and Claude series), highlighting huge room for improvement in multilingual code debugging scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。