BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
Monash University(莫纳什大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61);Singapore Management University(新加坡管理大学);Detomo Inc., Japan(日本Detomo公司);Queen Mary University of London(伦敦玛丽女王大学);University of Notre Dame(圣母大学);TU Darmstadt(达姆施塔特工业大学);Independent(独立研究者);University of Virginia(弗吉尼亚大学);Inria(法国国家信息与自动化研究所);Intel(英特尔公司);Tencent AI Lab(腾讯AI Lab);Cornell University(康奈尔大学);University College London(伦敦大学学院);UNC Chapel Hill(北卡罗来纳大学教堂山分校);UC Berkeley(加州大学伯克利分校);MIT(麻省理工学院);Shanghai Jiaotong University(上海交通大学);Uber(优步公司);UIUC(伊利诺伊大学厄巴纳-香槟分校);Sea AI Lab(Sea AI实验室);AWS AI Labs(亚马逊云科技AI实验室);Contextual AI(Contextual AI公司);Carnegie Mellon University(卡内基梅隆大学);ServiceNow Research(ServiceNow研究院);Hugging Face(Hugging Face公司)
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Ch… 展开作者
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra
备注:
Accpeted at ICLR 2025 (Oral), built with love by the BigCode community:)
Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for this http URL assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.
Max Planck Institute for Informatics Saarland Campus(马克斯·普朗克信息研究所萨尔兰州分校);Computer Science Saarland University(萨尔兰州大学计算机科学系);Max Planck Institute for Software Systems Saarland Campus(马克斯·普朗克软件系统研究所萨尔兰州分校)
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
MATS;Mila – Quebec AI Institute(Mila – 魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Astra Fellowship;ELLIS Institute Tübingen, MPI for Intelligent Systems & Tübingen AI Center;LawZero;Google DeepMind(谷歌DeepMind)
Benchmarking Complex Instruction-Following with Multiple Constraints Composition
Tsinghua University(清华大学);University of Electronic Science and Technology of China(电子科技大学);Zhipu AI(智谱AI);China University of Geosciences(中国地质大学);Central China Normal University(华中师范大学)
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);Zhipu AI(智谱AI);University of Electronic Science and Technology of China(电子科技大学)
ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following
Graduate School of Informatics, Kyoto University, Japan(京都大学信息科学研究生院);College of Computer Science and Artificial Intelligence, Fudan University, China(复旦大学计算机科学与人工智能学院);Graduate School of Engineering, Kyoto University, Japan(京都大学工学研究科)
Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks
Institute of Mathematics and Computer Science, University of São Paulo(圣保罗大学数学与计算机科学学院);Institute of Informatics, Federal University of Goiás(戈亚斯联邦大学信息学院);HUG Labs(HUG实验室);Advanced Knowledge Center for Immersive Technologies (AKCIT)(沉浸式技术高级知识中心)
SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Fudan University(复旦大学);Tsinghua University(清华大学);Beihang University(北航大学)
Can Multimodal LLMs See Science Instruction? Benchmarking Pedagogical Reasoning in K-12 Classroom Videos
Department of Computer Science, Drexel University(德雷塞尔大学计算机科学系);Department of Teaching and Learning, Washington State University(华盛顿州立大学教学与学习系);College of Chemistry, Beijing Normal University(北京师范大学化学学院);Department of Computer Science, University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校计算机科学系);Department of Data Science, City University of Hong Kong(香港城市大学数据科学系)
ReFormBench: Benchmarking the Stability of Response-Conditioned Format Correction in Large Language Models
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);College of Software, Jilin University(吉林大学软件学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(中国电信人工智能技术(北京)有限公司星辰AGI实验室);BiliBili Inc.(哔哩哔哩股份有限公司);University of Ottawa(渥太华大学);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(吉林大学知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);The Hong Kong University of Science and Technology(香港科技大学);Beihang University(北京航空航天大学);University of the Chinese Academy of Sciences(中国科学院大学);ETH Zürich(苏黎世联邦理工学院);The Hong Kong Polytechnic University(香港理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shanghai AI Lab(上海人工智能实验室)
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Hong Kong University of Science and Technology(香港科学与技术大学);DAMO Academy, Alibaba Group(阿里云达摩院)