arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.02037cs.SEcs.CL
收藏

EffiBench:基准测试自动生成代码的效率

EffiBench: Benchmarking the Efficiency of Automatically Generated Code

The University of Hong Kong(香港大学);University of Waterloo(滑铁卢大学);Shanghai AI Laboratory(上海人工智能实验室)

Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, Jie M. Zhang

AI总结:

本文提出含 1,000 个效率关键问题的 EffiBench 基准,评测 42 个 LLM 后发现其生成代码效率普遍低于人工 SOTA 方案。

摘要:

代码生成模型已日益成为辅助软件开发不可或缺的一部分。尽管当前研究已经彻底考察了代码生成模型所生成代码的正确性,但在绿色计算和可持续发展工作中发挥关键作用的一个重要方面却常常被忽视。本文提出了 EffiBench,这是一个包含 1,000 个效率关键型编程问题的基准,用于评估代码生成模型生成代码的效率。EffiBench 包含多样化的 LeetCode 编程问题集合。每个问题都配有一个可执行的、人工编写的标准解决方案,该方案在 LeetCode 解决方案排行榜上取得了 SOTA 效率。借助 EffiBench,我们实证考察了 42 个大型语言模型(35 个开源模型和 7 个闭源模型)生成高效代码的能力。我们的评估结果表明,LLM 生成代码的效率通常低于人工编写的标准解决方案。例如,GPT-4 生成代码的平均执行时间是人工编写标准解决方案的 3.12 倍。在最极端的情况下,GPT-4 生成代码的执行时间和总内存使用量分别达到标准解决方案的 13.89 倍和 43.92 倍。EffiBench 的源代码发布在 https://github.com/huangd1999/EffiBench。我们还在 https://huggingface.co/spaces/EffiBench/effibench-leaderboard 提供了 LeaderBoard。

英文摘要:

Code generation models have increasingly become integral to aiding software development. Although current research has thoroughly examined the correctness of the code produced by code generation models, a vital aspect that plays a pivotal role in green computing and sustainability efforts has often been neglected. This paper presents EffiBench, a benchmark with 1,000 efficiency-critical coding problems to assess the efficiency of code generated by code generation models. EffiBench contains a diverse set of LeetCode coding problems. Each problem is paired with an executable human-written canonical solution, which obtains the SOTA efficiency on the LeetCode solution leaderboard. With EffiBench, we empirically examine the ability of 42 large language models (35 open-source and 7 closed-source) to generate efficient code. Our evaluation results demonstrate that the efficiency of the code generated by LLMs is generally worse than the efficiency of human-written canonical solutions. For example, GPT-4 generated code has an average \textbf{3.12} times execution time that of the human-written canonical solutions. In the most extreme cases, the execution time and total memory usage of GPT-4 generated code are \textbf{13.89} and \textbf{43.92} times that of the canonical solutions. The source code of EffiBench is released on https://github.com/huangd1999/EffiBench. We also provide the LeaderBoard at https://huggingface.co/spaces/EffiBench/effibench-leaderboard.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑