arXiv 2311.085882024-06-07cs.CLcs.AIcs.SE

CodeScope：一个基于执行的多语言多任务多维基准，用于评估大语言模型在代码理解与生成上的能力

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation

University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Alibaba Group（阿里巴巴集团）；TU Wien（维也纳技术大学）；University of Chinese Academy of Sciences（中国科学院大学）

Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng

备注：: Accepted by ACL 2024 main conference

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CodeScope基准，覆盖43种语言和8类编码任务，从长度、难度和效率三个维度并借助MultiCodeEngine执行引擎，更全面、真实地评估大语言模型的代码理解与生成能力。

摘要：

大语言模型（LLMs）在辅助人类编程和促进编程自动化方面已展现出卓越性能。然而，现有用于评估LLMs代码理解与生成能力的基准存在严重局限。首先，大多数基准不够充分，因为它们只关注少数流行编程语言和特定任务，而现实世界的软件开发场景迫切需要实现具有多语言和多任务编程环境的系统，以满足多样化需求。其次，大多数基准未能考虑生成代码的实际可执行性以及执行结果的一致性。为了弥合现有基准与实际应用期望之间的差距，我们引入了CodeScope，这是一个基于执行、多语言、多任务、多维度的评估基准，用于全面衡量LLMs在编码任务上的能力。CodeScope覆盖43种编程语言和八类编码任务。它从三个维度（视角）评估LLMs的编码性能：长度、难度和效率。为便于对代码生成进行基于执行的评估，我们开发了MultiCodeEngine，这是一个支持14种编程语言的自动化代码执行引擎。最后，我们系统地评估和分析了八个主流LLMs，并证明了与其他基准相比，CodeScope在评估LLMs代码理解与生成任务方面具有更优越的广度和挑战性。CodeScope基准和代码已在 https://github.com/WeixiangYAN/CodeScope 公开提供。

英文摘要：

Large Language Models (LLMs) have demonstrated remarkable performance on assisting humans in programming and facilitating programming automation. However, existing benchmarks for evaluating the code understanding and generation capacities of LLMs suffer from severe limitations. First, most benchmarks are insufficient as they focus on a narrow range of popular programming languages and specific tasks, whereas real-world software development scenarios show a critical need to implement systems with multilingual and multitask programming environments to satisfy diverse requirements. Second, most benchmarks fail to consider the actual executability and the consistency of execution results of the generated code. To bridge these gaps between existing benchmarks and expectations from practical applications, we introduce CodeScope, an execution-based, multilingual, multitask, multidimensional evaluation benchmark for comprehensively measuring LLM capabilities on coding tasks. CodeScope covers 43 programming languages and eight coding tasks. It evaluates the coding performance of LLMs from three dimensions (perspectives): length, difficulty, and efficiency. To facilitate execution-based evaluations of code generation, we develop MultiCodeEngine, an automated code execution engine that supports 14 programming languages. Finally, we systematically evaluate and analyze eight mainstream LLMs and demonstrate the superior breadth and challenges of CodeScope for evaluating LLMs on code understanding and generation tasks compared to other benchmarks. The CodeScope benchmark and code are publicly available at https://github.com/WeixiangYAN/CodeScope.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。