arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.06647cs.SEcs.AIcs.LG
收藏

LLM生成代码的效率如何?一个严谨且高标准的基准测试

How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark

University of Illinois Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);Qualcomm AI Research(高通人工智能研究部门)

Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, Hanghang Tong

AI总结:

本研究针对LLM生成代码效率评估缺失的问题,开发ENAMEL基准测试,提出eff@k指标及对应估计器,设计专家级参考方案与测试用例,评估发现LLM在生成高效代码上存在不足,原因是其难以设计高级算法且缺乏实现优化意识。

摘要:

大型语言模型(LLMs)的出现显著推动了程序合成的前沿发展。基于LLM的程序合成技术的进步需要对LLM生成代码进行全面评估。大多数评估框架聚焦于生成代码的(功能)正确性;而效率作为代码质量的重要衡量标准,在现有评估中被忽视。在本研究中,我们开发了ENAMEL(EfficeNcy AutoMatic EvaLuator,效率自动评估器),一个用于评估LLMs生成高效代码能力的严谨且高标准的基准测试。首先,我们提出了一种名为eff@k的新效率指标,它将正确性的pass@k指标推广到效率领域,并适当地处理右删失执行时间。此外,我们通过Rao-Blackwell化推导了eff@k的无偏且方差降低的估计器;我们还为新估计器提供了数值稳定的实现。其次,为了设定效率评估的高标准,我们聘请人类专家设计最佳算法和实现作为我们的效率参考解决方案,其中许多方案比HumanEval和HumanEval+中现有的标准解决方案高效得多。此外,为确保评估的严谨性,我们聘请人类专家精心设计强大的测试用例生成器,以过滤错误代码并区分次优算法。使用我们的ENAMEL基准测试对30个流行LLMs进行的广泛研究表明,LLMs在生成专家级高效代码方面仍有不足。通过我们问题集的两个子集,我们证明这种不足是因为当前LLMs难以设计高级算法,且几乎没有实现优化的意识。我们的基准测试公开可访问:https://github.com/q-rz/enamel。

英文摘要:

The emergence of large language models (LLMs) has significantly pushed the frontiers of program synthesis. Advancement of LLM-based program synthesis calls for a thorough evaluation of LLM-generated code. Most evaluation frameworks focus on the (functional) correctness of generated code; efficiency, as an important measure of code quality, has been overlooked in existing evaluations. In this work, we develop ENAMEL (EfficeNcy AutoMatic EvaLuator), a rigorous and high-standard benchmark for evaluating the capability of LLMs in generating efficient code. Firstly, we propose a new efficiency metric called eff@k, which generalizes the pass@k metric from correctness to efficiency and appropriately handles right-censored execution time. Furthermore, we derive an unbiased and variance-reduced estimator of eff@k via Rao--Blackwellization; we also provide a numerically stable implementation for the new estimator. Secondly, to set a high-standard for efficiency evaluation, we employ a human expert to design best algorithms and implementations as our reference solutions of efficiency, many of which are much more efficient than existing canonical solutions in HumanEval and HumanEval+. Moreover, to ensure a rigorous evaluation, we employ a human expert to curate strong test case generators to filter out wrong code and differentiate suboptimal algorithms. An extensive study across 30 popular LLMs using our benchmark ENAMEL shows that LLMs still fall short of generating expert-level efficient code. Using two subsets of our problem set, we demonstrate that such deficiency is because current LLMs struggle in designing advanced algorithms and are barely aware of implementation optimization. Our benchmark is publicly available at https://github.com/q-rz/enamel .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑