arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.09059cs.SE
收藏

评估大语言模型生成的代码:一个基准和开发者研究

Evaluating LLM-Generated Code: A Benchmark and Developer Study

IU Internationale Hochschule Bad Honnef(德国国际高等教育学院)

Joanna Szych, Anne Schwerk

AI总结:

本文提出了一种定制的树折评估方法,用于评估大语言模型生成的代码,弥补了现有基准在代码质量和可操作性方面的不足,并通过对比三个通用大语言模型展示了其有效性。

摘要:

代码生成是大型语言模型广泛应用和高度成功的任务之一。鉴于其受欢迎程度,有许多专门针对代码生成的基准可以帮助选择最佳模型。然而,这些基准主要关注解决方案的正确性,忽略了其他方面,如代码质量和可操作性。本文旨在描述一种定制的树折评估方法,用于评估由大型语言模型生成的代码,以弥补这一差距。该方法包括基于复杂多级计算机科学项目的专用正确性基准、代码质量验证以及通过结构化代码审查过程收集的开发者对生成代码样本意见的调查。所提出的方法的使用和有效性通过评估和比较三个通用大型语言模型:GPT-4.1、DeepSeek-V3-0324和Claude Opus 4来展示。结果表明,通过开发者收集的审查可以得出许多新的发现,特别是与代码处于生产就绪状态相关的发现,这些在使用标准正确性导向基准方法时是无法获得的。

英文摘要:

Code generation is one of the tasks for which the use of Large Language Models is widely adopted and highly successful. Given this popularity, there are many benchmarks dedicated to code generation that can help select the best model. However, they primarily focus on measuring solution correctness, leaving other aspects, such as code quality and usability, behind. This paper aims to describe a custom tree-fold evaluation methodology for code generated by Large Language Models that bridges this gap. The methodology includes a dedicated correctness benchmark based on a complex multi-level computer science project, code quality verification, and a survey of developers' opinions on generated code samples gathered through a structured code-review process. The proposed methodology's usage and usefulness are demonstrated by evaluating and comparing three general-purpose Large Language Models: GPT-4.1, DeepSeek-V3-0324, and Claude Opus 4. The results show that reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑