评估代码语言模型时,功能正确性就足够了吗?探索生成代码的多样性
Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes
AI总结:
本研究提出了一种系统评估代码语言模型生成代码多样性的方法,引入基于大型模型的成对相似度度量,发现当前模型倾向于生成功能正确但多样性有限的代码。
摘要:
语言模型(LMs)在根据自然语言需求生成代码方面展现出了令人印象深刻的能力。在本研究中,我们强调LMs生成代码的多样性作为评估其代码生成能力的关键标准,以补充功能正确性。尽管具有实际意义,但目前缺乏专注于评估生成代码多样性的研究,这忽视了其在代码LMs发展中的重要性。我们提出了一种系统的方法来评估生成代码的多样性,利用各种指标来衡量代码间相似度以及功能正确性。具体而言,我们引入了一种成对代码相似度度量方法,利用大型LMs在代码理解和推理方面的能力,该方法展现出与人类判断的最高相关性。我们广泛调查了各种因素对生成代码质量的影响,包括模型大小、温度、训练方法、提示策略和输入问题的难度。我们对测试通过分数和代码间相似度分数之间正相关的一致观察表明,当前的LMs倾向于生成功能正确但多样性有限的代码。
英文摘要:
Language models (LMs) have exhibited impressive abilities in generating codes from natural language requirements. In this work, we highlight the diversity of code generated by LMs as a critical criterion for evaluating their code generation capabilities, in addition to functional correctness. Despite its practical implications, there is a lack of studies focused on assessing the diversity of generated code, which overlooks its importance in the development of code LMs. We propose a systematic approach to evaluate the diversity of generated code, utilizing various metrics for inter-code similarity as well as functional correctness. Specifically, we introduce a pairwise code similarity measure that leverages large LMs' capabilities in code understanding and reasoning, demonstrating the highest correlation with human judgment. We extensively investigate the impact of various factors on the quality of generated code, including model sizes, temperatures, training approaches, prompting strategies, and the difficulty of input problems. Our consistent observation of a positive correlation between the test pass score and the inter-code similarity score indicates that current LMs tend to produce functionally correct code with limited diversity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。