我们提出了差异性能评估(Differential Performance Evaluation,DPE)框架,旨在可靠评估大语言模型(Large Language Models,LLMs)的高效代码生成能力。传统代码基准依赖简单的测试输入,且缺乏有效的复合指标,往往无法提供关于代码效率的可靠结论。DPE通过聚焦对效率要求高的编程任务,并建立一套有洞察力的性能评估复合指标来解决这些问题。\nDPE分为两个阶段运行:为了构建效率数据集,它从现有代码基准中筛选对效率要求高的任务,并生成计算开销大的输入,以测试LLM生成方案的效率。为评估代码效率,DPE对新生成的方案进行性能分析,并将其与一组呈现不同效率水平的参考方案进行全局比较,匹配到的水平即为其效率得分。\n作为概念验证,我们使用DPE构建了EvalPerf基准,该基准包含121个具有性能挑战性的编码任务。我们的综合评估得出了关于模型规模、指令微调和提示工程对效率影响的有趣结论。例如,虽然缩放定律无法解释代码效率,但通用指令微调对代码正确性和效率都有提升。我们还通过检验DPE的有效性对该评估方法本身进行了评估,结果表明EvalPerf即使跨平台使用也具备可靠性和便捷性。
英文摘要:
We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.
CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts
Department of Computer Science(计算机科学系);Chandigarh University Punjab(昌迪加尔大学旁遮普);Department of Technology Management and Innovation(技术管理与创新系);New York University(纽约大学)
CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Multimodal Art Projection Research Community(多模态艺术投影研究社区);University of Waterloo(滑铁卢大学);HKUST(香港科技大学);University of Manchester(曼彻斯特大学);Tongji University(同济大学);Vector Institute(矢量研究所)
Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学教育部计算语言学重点实验室)
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,计算机学院,北京大学);Microsoft Research Asia(微软亚洲研究院)
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)
Evaluating Large Language Models on Controlled Generation Tasks
University of Southern California(南加州大学);University of California, Los Angeles(加州大学洛杉矶分校);Amazon(亚马逊);Google DeepMind(谷歌DeepMind);ETH Zurich(苏黎世联邦理工学院)
State Key Lab of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);Key Lab of AI Safety, Chinese Academy of Sciences(人工智能安全重点实验室,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Models
Computational Linguistics, Department of Linguistics University of Potsdam(波恩大学语言学系);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)