arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.09043cs.SE
收藏

CodeScore:通过学习代码执行来评估代码生成

CodeScore: Evaluating Code Generation by Learning Code Execution

Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, Zhi Jin

AI总结:

本文提出基于大语言模型的代码评估指标CodeScore及统一学习框架UniCE,通过学习代码执行来评估生成代码的功能正确性,在三种输入格式上显著提升了与功能正确性的相关性。

摘要:

一个合适的代码评估指标(CEM)深刻影响着代码生成的发展,而代码生成是自然语言处理与软件工程中的一个重要研究领域。当前主流的基于匹配的CEM(如BLEU、Accuracy和CodeBLEU)存在两个显著缺陷。1. 它们主要衡量代码之间的表面差异,而不考虑其功能等价性。然而,功能等价性在评估代码生成的有效性时至关重要,因为不同的代码可以执行相同的操作。2. 它们主要为Ref-only输入格式设计。然而,代码评估需要输入格式的多样性。除了Ref-only,还有NL-only和Ref&NL格式,而现有的基于匹配的CEM无法有效适应这些格式。在本文中,我们提出了CodeScore,一个基于大语言模型(LLM)的CEM,用于估计生成代码在三种输入类型上的功能正确性。为了获得CodeScore,我们提出了UniCE,一个统一的代码生成学习框架,让LLM在统一输入下学习代码执行(即学习生成代码的PassRatio和Executability)。在多个代码评估数据集上的大量实验结果表明,与其他CEM相比,CodeScore与功能正确性的相关性绝对提升了最高58.87%,达到了最先进的性能,并能有效处理三种输入格式。

英文摘要:

A proper code evaluation metric (CEM) profoundly impacts the evolution of code generation, which is an important research field in NLP and software engineering. Prevailing match-based CEMs (e.g., BLEU, Accuracy, and CodeBLEU) suffer from two significant drawbacks. 1. They primarily measure the surface differences between codes without considering their functional equivalence. However, functional equivalence is pivotal in evaluating the effectiveness of code generation, as different codes can perform identical operations. 2. They are predominantly designed for the Ref-only input format. However, code evaluation necessitates versatility in input formats. Aside from Ref-only, there are NL-only and Ref\&NL formats, which existing match-based CEMs cannot effectively accommodate. In this paper, we propose CodeScore, a large language model (LLM)-based CEM, which estimates the functional correctness of generated code on three input types. To acquire CodeScore, we present UniCE, a unified code generation learning framework, for LLMs to learn code execution (i.e., learning PassRatio and Executability of generated code) with unified input. Extensive experimental results on multiple code evaluation datasets demonstrate that CodeScore absolutely improves up to 58.87% correlation with functional correctness compared to other CEMs, achieves state-of-the-art performance, and effectively handles three input formats.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑