The emergence of large language models (LLMs) has significantly pushed the frontiers of program synthesis. Advancement of LLM-based program synthesis calls for a thorough evaluation of LLM-generated code. Most evaluation frameworks focus on the (functional) correctness of generated code; efficiency, as an important measure of code quality, has been overlooked in existing evaluations. In this work, we develop ENAMEL (EfficeNcy AutoMatic EvaLuator), a rigorous and high-standard benchmark for evaluating the capability of LLMs in generating efficient code. Firstly, we propose a new efficiency metric called eff@k, which generalizes the pass@k metric from correctness to efficiency and appropriately handles right-censored execution time. Furthermore, we derive an unbiased and variance-reduced estimator of eff@k via Rao--Blackwellization; we also provide a numerically stable implementation for the new estimator. Secondly, to set a high-standard for efficiency evaluation, we employ a human expert to design best algorithms and implementations as our reference solutions of efficiency, many of which are much more efficient than existing canonical solutions in HumanEval and HumanEval+. Moreover, to ensure a rigorous evaluation, we employ a human expert to curate strong test case generators to filter out wrong code and differentiate suboptimal algorithms. An extensive study across 30 popular LLMs using our benchmark ENAMEL shows that LLMs still fall short of generating expert-level efficient code. Using two subsets of our problem set, we demonstrate that such deficiency is because current LLMs struggle in designing advanced algorithms and are barely aware of implementation optimization. Our benchmark is publicly available at https://github.com/q-rz/enamel .
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution
University of Oslo(奥斯陆大学);Technology Innovation Institute(技术创新研究所);Eötvös Lóránd University(欧多西亚·洛拉恩大学);Guelma University(盖尔马大学);University of Oslo, Cyentific AS(奥斯陆大学,Cyentific AS)
Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
University of Maryland(马里兰大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);UK AI Security Institute(英国人工智能安全研究所);University of Toronto(多伦多大学);Northeastern University(东北大学);Haize Labs(Haize实验室);Dalhousie University(达尔豪斯大学);University of Waterloo(滑铁卢大学);University of Oxford(牛津大学);MIT(麻省理工学院)
MCCoder: Streamlining Motion Control with LLM-Assisted Code Generation and Rigorous Verification
Thrust of Data Science and Analytics, The Hong Kong University of Science and Technology (Guangzhou)(数据科学与分析 thrust,香港科学与技术大学(广州));Department of Industrial Engineering and Decision Analytics, The Hong Kong University of Science and Technology(工业工程与决策分析系,香港科学与技术大学);MOVENSYS Inc.(MOVENSYS公司);University of Cologne(科隆大学)
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
University of Maryland(马里兰大学);Carnegie Mellon University(卡内基梅隆大学);New York University(纽约大学);ELLIS Institute Tübingen, MPI Intelligent Systems(埃利斯研究所蒂宾根分部、马克斯·普朗克智能系统研究所)
Rutgers University(罗格斯大学);University of Pittsburgh(匹兹堡大学);Korea University(高丽大学);Korean National Police University(韩国警察大学);Ewha Womans University(梨花女子大学)
Numbers Beat Words: A Rigorous On-Premise Benchmark for Coupled MIMO Controller Tuning
Jinling Institute of Technology(金陵科技学院);College of Water Resources and Civil Engineering, China Agricultural University(中国农业大学水利与土木工程学院);Zhejiang University(浙江大学)
"I know myself better, but not really greatly": How Well Can LLMs Detect and Explain LLM-Generated Texts?
School of Cyber Science and Engineering, Shanghai Jiao Tong University, China(上海交通大学信息科学与工程学院);Department of Computing Science, University of Aberdeen, UK(阿伯丁大学计算机科学系);Institute of Cyber Security for Society (iCSS) & School of Computing, University of Kent, UK(社会网络安全研究所(iCSS)及肯特大学计算机学院)