arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.06450cs.SEcs.CLcs.LG
收藏

面向高效代码生成的语言模型评估

Evaluating Language Models for Efficient Code Generation

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Tongji University(同济大学)

Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, Lingming Zhang

AI总结:

本研究提出DPE框架以可靠评估大语言模型的高效代码生成能力,构建含121个性能挑战性任务的EvalPerf基准,揭示了模型规模、指令微调等对代码效率的影响,验证了该评估方法的可靠性与跨平台便捷性。

摘要:

我们提出了差异性能评估(Differential Performance Evaluation,DPE)框架,旨在可靠评估大语言模型(Large Language Models,LLMs)的高效代码生成能力。传统代码基准依赖简单的测试输入,且缺乏有效的复合指标,往往无法提供关于代码效率的可靠结论。DPE通过聚焦对效率要求高的编程任务,并建立一套有洞察力的性能评估复合指标来解决这些问题。\nDPE分为两个阶段运行:为了构建效率数据集,它从现有代码基准中筛选对效率要求高的任务,并生成计算开销大的输入,以测试LLM生成方案的效率。为评估代码效率,DPE对新生成的方案进行性能分析,并将其与一组呈现不同效率水平的参考方案进行全局比较,匹配到的水平即为其效率得分。\n作为概念验证,我们使用DPE构建了EvalPerf基准,该基准包含121个具有性能挑战性的编码任务。我们的综合评估得出了关于模型规模、指令微调和提示工程对效率影响的有趣结论。例如,虽然缩放定律无法解释代码效率,但通用指令微调对代码正确性和效率都有提升。我们还通过检验DPE的有效性对该评估方法本身进行了评估,结果表明EvalPerf即使跨平台使用也具备可靠性和便捷性。

英文摘要:

We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑