Code generation models have increasingly become integral to aiding software development. Although current research has thoroughly examined the correctness of the code produced by code generation models, a vital aspect that plays a pivotal role in green computing and sustainability efforts has often been neglected. This paper presents EffiBench, a benchmark with 1,000 efficiency-critical coding problems to assess the efficiency of code generated by code generation models. EffiBench contains a diverse set of LeetCode coding problems. Each problem is paired with an executable human-written canonical solution, which obtains the SOTA efficiency on the LeetCode solution leaderboard. With EffiBench, we empirically examine the ability of 42 large language models (35 open-source and 7 closed-source) to generate efficient code. Our evaluation results demonstrate that the efficiency of the code generated by LLMs is generally worse than the efficiency of human-written canonical solutions. For example, GPT-4 generated code has an average \textbf{3.12} times execution time that of the human-written canonical solutions. In the most extreme cases, the execution time and total memory usage of GPT-4 generated code are \textbf{13.89} and \textbf{43.92} times that of the canonical solutions. The source code of EffiBench is released on https://github.com/huangd1999/EffiBench. We also provide the LeaderBoard at https://huggingface.co/spaces/EffiBench/effibench-leaderboard.
Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education(教育部新一代人工智能技术及其交叉应用关键实验室(东南大学));The University of Manchester(曼彻斯特大学);The University of Edinburgh(爱丁堡大学);Law and Innovation Lab, School of Law, Southeast University(东南大学法学院创新与法律实验室);State Key Lab. for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);The Chinese University of Hong Kong(香港中文大学);Monash University(莫纳什大学)
Interdisciplinary Program in Artificial Intelligence, Seoul National University(人工智能交叉学科项目,首尔国立大学);Electronics and Telecommunications Research Institute(电子与电信研究所)
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
University of International Relations(国际关系学院);Kedge Business School(凯致商学院);Peking University(北京大学);Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);Capital Normal University(首都师范大学)
BAGEL: Benchmarking Animal Knowledge Expertise in Language Models
Earth Species Project(地球物种项目);NYU Center for Data Science(纽约大学数据科学中心;纽约大学上海);NYU Shanghai(纽约大学上海数据科学中心;纽约大学-复旦大学数学科学研究所);NYU Shanghai Center for Data Science;NYU-ECNU Institute of Mathematical Sciences at NYU Shanghai
AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge
Nanyang Technological University(南洋理工大学);University of California, Santa Barbara(加州大学圣芭芭拉分校);National University of Singapore(新加坡国立大学);Shanghai Jiao Tong University(上海交通大学);University of Arizona(亚利桑那大学)
An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Department of Computer Science University of Illinois Urbana–Champaign(计算机科学系伊利诺伊大学厄巴纳-香槟分校);Department of Computer Science Stanford University(计算机科学系斯坦福大学)
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Monash University(莫纳什大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61);Singapore Management University(新加坡管理大学);Detomo Inc., Japan(日本Detomo公司);Queen Mary University of London(伦敦玛丽女王大学);University of Notre Dame(圣母大学);TU Darmstadt(达姆施塔特工业大学);Independent(独立研究者);University of Virginia(弗吉尼亚大学);Inria(法国国家信息与自动化研究所);Intel(英特尔公司);Tencent AI Lab(腾讯AI Lab);Cornell University(康奈尔大学);University College London(伦敦大学学院);UNC Chapel Hill(北卡罗来纳大学教堂山分校);UC Berkeley(加州大学伯克利分校);MIT(麻省理工学院);Shanghai Jiaotong University(上海交通大学);Uber(优步公司);UIUC(伊利诺伊大学厄巴纳-香槟分校);Sea AI Lab(Sea AI实验室);AWS AI Labs(亚马逊云科技AI实验室);Contextual AI(Contextual AI公司);Carnegie Mellon University(卡内基梅隆大学);ServiceNow Research(ServiceNow研究院);Hugging Face(Hugging Face公司)