arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.16601cs.SEcs.AI
收藏

对大型语言模型生成代码的检查

Examination of Code generated by Large Language Models

Technical University of Applied Sciences Augsburg(奥格斯堡应用技术大学);Linnaeus University(林奈大学)

Robin Beer, Alexander Feix, Tim Guttzeit, Tamara Muras, Vincent Müller, Maurice Rauscher, Florian Schäffler, Welf Löwe

AI总结:

本文通过受控实验评估了ChatGPT和Copilot在Java和Python中生成算法及单元测试的正确性与覆盖率,发现模型、语言、代码类型及时间推移间存在显著差异,并提供了可重复的评估方法。

摘要:

大型语言模型(LLMs),如ChatGPT和Copilot,正通过自动化代码生成来变革软件开发,并且可以说能够实现快速原型设计、支持教育并提升生产力。因此,生成代码的正确性和质量应当与手写代码相当。为了评估LLMs在生成高质量正确代码方面的现状,我们使用ChatGPT和Copilot进行了受控实验:我们让这些LLMs用Java和Python生成简单算法以及相应的单元测试,并评估了生成(测试)代码的正确性和质量(覆盖率)。我们观察到不同LLMs之间、不同语言之间、算法与测试代码之间以及随着时间推移都存在显著差异。本文报告了这些结果以及实验方法,从而允许随着时间的推移对更多算法、语言和LLMs进行可重复且可比较的评估。

英文摘要:

Large language models (LLMs), such as ChatGPT and Copilot, are transforming software development by automating code generation and, arguably, enable rapid prototyping, support education, and boost productivity. Therefore, correctness and quality of the generated code should be on par with manually written code. To assess the current state of LLMs in generating correct code of high quality, we conducted controlled experiments with ChatGPT and Copilot: we let the LLMs generate simple algorithms in Java and Python along with the corresponding unit tests and assessed the correctness and the quality (coverage) of the generated (test) codes. We observed significant differences between the LLMs, between the languages, between algorithm and test codes, and over time. The present paper reports these results together with the experimental methods allowing repeated and comparable assessments for more algorithms, languages, and LLMs over time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑