Amidst the recent strides in evaluating Large Language Models for Code (Code LLMs), existing benchmarks have mainly focused on the functional correctness of generated code, neglecting the importance of their computational efficiency. To fill the gap, we present Mercury, the first code efficiency benchmark for Code LLMs. It comprises 1,889 Python tasks, each accompanied by adequate solutions that serve as real-world efficiency baselines, enabling a comprehensive analysis of the runtime distribution. Based on the distribution, we introduce a new metric Beyond, which computes a runtime-percentile-weighted Pass score to reflect functional correctness and code efficiency simultaneously. On Mercury, leading Code LLMs can achieve 65% on Pass, while less than 50% on Beyond. Given that an ideal Beyond score would be aligned with the Pass score, it indicates that while Code LLMs exhibit impressive capabilities in generating functionally correct code, there remains a notable gap in their efficiency. Finally, our empirical experiments reveal that Direct Preference Optimization (DPO) serves as a robust baseline for enhancing code efficiency compared with Supervised Fine Tuning (SFT), which paves a promising avenue for future exploration of efficient code generation. Our code and data are available on GitHub: https://github.com/Elfsong/Mercury.
Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs
Computer Systems Department, Jožef Stefan Institute, Ljubljana, Slovenia(计算机系统部,乔塞夫·斯塔芬研究所,卢布尔雅那,斯洛文尼亚);Jožef Stefan International Postgraduate School, Ljubljana, Slovenia(乔塞夫·斯塔芬国际研究生学院,卢布尔雅那,斯洛文尼亚);Center for Astrophysics and Cosmology, University of Nova Gorica, Nova Gorica, Slovenia(天体物理与宇宙学中心,诺瓦戈里察大学,诺瓦戈里察,斯洛文尼亚)
AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Northeastern University(东北大学);Carnegie Mellon University(卡内基梅隆大学);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Sun Yat-sen University(中山大学)
Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
Huazhong University of Science and Technology(华中科技大学);Griffith University(格里菲斯大学);Squirrel Ai Learning(squirrel ai 学习);Shanghai Jiaotong University(上海交通大学);Lehigh University(莱斯大学)
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
Shanghai Institute of AI for Education(上海人工智能教育研究院);School of Computer Science(计算机科学学院);East China Normal University(东华大学);Tencent Inc.(腾讯公司);Shanghai Innovation Institute(上海创新研究院)
Chiseling Out Efficiency: Structured Skeleton Supervision for Efficient Code Generation
Shandong Normal University(山东师范大学);Nanjing University(南京大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);Chongqing University(重庆大学);Institute of Information Engineering at Chinese Academy of Sciences(中国科学院信息工程研究所);Peking University(北京大学)
Benchmarking Energy Efficiency of Large Language Models Using vLLM
Allumni Master Applied IT Fontys University of Applied Sciences Eindhoven, The Netherlands;Sustainable Data \& AI Group Fontys University of Applied Sciences Eindhoven, The Netherlands
Matching Matters: A Fair Quality-Efficiency Benchmark for Command-Line Agents
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);Hangzhou Institute for Advanced Study, Chinese Academy of Sciences(中国科学院杭州高等研究院)