arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.15877cs.SEcs.AIcs.CL
收藏

BigCodeBench:基于多样函数调用和复杂指令的代码生成基准测试

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Monash University(莫纳什大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61);Singapore Management University(新加坡管理大学);Detomo Inc., Japan(日本Detomo公司);Queen Mary University of London(伦敦玛丽女王大学);University of Notre Dame(圣母大学);TU Darmstadt(达姆施塔特工业大学);Independent(独立研究者);University of Virginia(弗吉尼亚大学);Inria(法国国家信息与自动化研究所);Intel(英特尔公司);Tencent AI Lab(腾讯AI Lab);Cornell University(康奈尔大学);University College London(伦敦大学学院);UNC Chapel Hill(北卡罗来纳大学教堂山分校);UC Berkeley(加州大学伯克利分校);MIT(麻省理工学院);Shanghai Jiaotong University(上海交通大学);Uber(优步公司);UIUC(伊利诺伊大学厄巴纳-香槟分校);Sea AI Lab(Sea AI实验室);AWS AI Labs(亚马逊云科技AI实验室);Contextual AI(Contextual AI公司);Carnegie Mellon University(卡内基梅隆大学);ServiceNow Research(ServiceNow研究院);Hugging Face(Hugging Face公司)
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Ch… 展开作者

Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, Leandro Von Werra

AI总结:

本文提出BigCodeBench基准,通过1,140个细粒度任务评估LLMs在139个库和7个领域中调用多样函数及遵循复杂指令的能力,发现当前模型表现远低于人类,凸显了该领域的改进需求。

摘要:

任务自动化通过大型语言模型(LLMs)借助Python代码得到了极大的推动,其应用范围涵盖从软件工程开发到通用推理的各类任务。尽管当前的基准测试表明,LLMs能够像人类开发者一样使用程序解决任务,但它们的大多数评估仅限于简短且自包含的算法任务或独立的函数调用。解决具有挑战性且实用的任务需要具备将多样函数调用作为工具的能力,以高效实现数据分析和Web开发等功能。此外,使用多种工具解决一项任务需要通过准确理解复杂指令来进行组合推理。同时满足这两个特性对LLMs构成了巨大挑战。为了评估LLMs通过程序解决具有挑战性且实用任务的能力,我们引入了BigCodeBench,这是一个基准测试,它要求LLMs针对1,140个细粒度任务,调用来自139个库和7个领域的多个函数调用作为工具。为了严格评估LLMs,每个任务平均包含5.6个测试用例,平均分支覆盖率达到99%。此外,我们提出了BigCodeBench的自然语言导向变体——BigCodeBench-Instruct,它自动将原始的docstrings转换为仅包含必要信息的简短指令。我们对60个LLMs的广泛评估表明,LLMs尚不具备遵循复杂指令以精确使用函数调用的能力,其得分最高仅为60%,显著低于人类97%的表现。这些结果强调了该领域需要进一步发展。

英文摘要:

Task automation has been greatly empowered by the recent advances in Large Language Models (LLMs) via Python code, where the tasks ranging from software engineering development to general-purpose reasoning. While current benchmarks have shown that LLMs can solve tasks using programs like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks or standalone function calls. Solving challenging and practical tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for this http URL assess how well LLMs can solve challenging and practical tasks via programs, we introduce BigCodeBench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks. To evaluate LLMs rigorously, each task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑