arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11326cs.SEcs.AI
收藏

GitHub Copilot:完美的代码补全工具?

GitHub Copilot: the perfect Code compLeeter?

KU Leuven(鲁汶大学);COSIC(COSIC研究院)

Ilja Siroš, Dave Singelée, Bart Preneel

AI总结:

该研究通过定制自动化框架在 LeetCode 上跨四种语言大规模评测 GitHub Copilot,分析其代码正确性、可靠性、效率及语言、难度和主题影响。

摘要:

本文旨在利用一个定制的自动化框架,基于 LeetCode 题库评估 GitHub Copilot 生成代码的质量。我们评估了 Copilot 在 4 种编程语言上的结果:Java、C++、Python3 和 Rust。我们的目标是评估 Copilot 在代码生成阶段的可靠性、生成代码的正确性,以及其对编程语言、题目难度级别和题目主题的依赖。除此之外,我们还评估代码的时间与内存效率,并将其与人类的平均结果进行比较。总体而言,我们为每种编程语言生成了 1760 道题的解答,并评估每道题的所有 Copilot 建议,最终在两个月的时间内向 LeetCode 提交了超过 50000 次。我们发现 Copilot 成功解决了大多数问题。然而,Copilot 在使用 Java 和 C++ 生成代码时比使用 Python3 和 Rust 更为成功。此外,就 Python3 而言,Copilot 在代码生成阶段被证明相当不可靠。我们还发现,Copilot 排名靠前的建议并不总是最佳的。另外,我们分析了题目的主题如何影响正确率。最后,根据来自 LeetCode 的统计信息,我们可以得出结论:Copilot 生成的代码比普通人更高效。

英文摘要:

This paper aims to evaluate GitHub Copilot's generated code quality based on the LeetCode problem set using a custom automated framework. We evaluate the results of Copilot for 4 programming languages: Java, C++, Python3 and Rust. We aim to evaluate Copilot's reliability in the code generation stage, the correctness of the generated code and its dependency on the programming language, problem's difficulty level and problem's topic. In addition to that, we evaluate code's time and memory efficiency and compare it to the average human results. In total, we generate solutions for 1760 problems for each programming language and evaluate all the Copilot's suggestions for each problem, resulting in over 50000 submissions to LeetCode spread over a 2-month period. We found that Copilot successfully solved most of the problems. However, Copilot was rather more successful in generating code in Java and C++ than in Python3 and Rust. Moreover, in case of Python3 Copilot proved to be rather unreliable in the code generation phase. We also discovered that Copilot's top-ranked suggestions are not always the best. In addition, we analysed how the topic of the problem impacts the correctness rate. Finally, based on statistics information from LeetCode, we can conclude that Copilot generates more efficient code than an average human.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑