arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.21579cs.SEcs.AI
收藏

LLM生成代码在Leetcode上的性能研究

A Performance Study of LLM-Generated Code on Leetcode

Univ. Lille(里尔大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所)

Tristan Coignion, Clément Quinton, Romain Rouvoy

AI总结:

本文通过Leetcode数据集评估18个LLM的代码生成效率,提出新的速度测量方法,发现不同LLM生成的代码性能相当且平均优于人类代码,为代码生成优化提供基础。

摘要:

本研究评估了大型语言模型(LLMs)生成代码的效率,并使用来自Leetcode的数据集衡量其与人工编写解决方案的性能对比。我们比较了18个LLMs,考虑了模型温度(model temperature)和成功率等因素,以及它们对代码性能的影响。本研究引入了一种新颖的方法来测量和比较LLM生成代码的速度,结果表明,无论采用哪种LLM,其生成的代码都具有可比的性能。我们还发现,LLM能够生成平均而言比人类编写的代码更高效的代码。本文进一步讨论了使用Leetcode作为基准数据集的问题、潜在数据污染所带来的局限性,以及该平台的测量可靠性。我们相信,我们的发现有助于更好地理解LLMs在代码生成方面的能力,并为该领域的未来优化奠定基础。

英文摘要:

This study evaluates the efficiency of code generation by Large Language Models (LLMs) and measures their performance against human-crafted solutions using a dataset from Leetcode. We compare 18 LLMs, considering factors such as model temperature and success rate, and their impact on code performance. This research introduces a novel method for measuring and comparing the speed of LLM-generated code, revealing that LLMs produce code with comparable performance, irrespective of the adopted LLM. We also find that LLMs are capable of generating code that is, on average, more efficient than the code written by humans. The paper further discusses the use of Leetcode as a benchmarking dataset, the limitations imposed by potential data contamination, and the platform's measurement reliability. We believe that our findings contribute to a better understanding of LLM capabilities in code generation and set the stage for future optimizations in the field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑