CodeScore-R:一种用于评估代码合成功能正确性的自动化鲁棒性指标
CodeScore-R: An Automated Robustness Metric for Assessing the FunctionalCorrectness of Code Synthesis
AI总结:
本文提出基于 UniXcoder 与对比学习的 CodeScore-R,无需测试用例即可评估代码功能正确性,并通过草图处理、语法等价变换和变异测试提升鲁棒性。
摘要:
评估指标在代码合成领域至关重要。常用的代码评估指标可分为三类:基于匹配的指标、基于语义的指标和基于执行的指标。其中,基于执行的 Pass@k 指标通过执行测试用例来准确评估预测代码的功能。然而,计算该指标需要大量开销,因此有必要设计一种无需测试用例即可评估预测代码功能的自动化评估指标。此外,一个优秀的评估指标应当具有鲁棒性,即即使预测代码发生微小变化,该指标仍能保持其准确性。为应对这些挑战,我们提出了一种基于 UniXcoder 和对比学习的自动化鲁棒指标 CodeScore-R,用于评估代码合成的功能。CodeScore-R 采用基于草图的处理、语法等价变换和变异测试等技术,有效缓解标识符、语法结构和运算符对评估结果的干扰。实验结果表明,在 Java 和 Python 的代码生成与代码迁移任务中,CodeScore-R 优于其他评估指标,并且与 Pass@k 指标更一致,同时表现出更强的鲁棒性。
英文摘要:
Evaluation metrics are crucial in the field of code synthesis. Commonly used code evaluation metrics canbe classified into three types: match-based, semantic-based, and execution-based. Among them, the execution-basedPass@k metric accurately assesses the functionality of predicted code by executing test cases. However, calculatingthis metric requires a significant amount of overhead, necessitating the design of an automated evaluation metric thatcan assess the functionality of predicted code without the need for test cases. Additionally, a good evaluation metricshould be robust, that is the metric can maintain its accuracy even when the predicted code undergoes minor changes.To address these challenges, we propose an automated robust metric, called CodeScore-R, based on UniXcoder andcontrastive learning, for evaluating the functionality of code synthesis. CodeScore-R employs techniques such assketch-based processing, syntactic-equivalent transformations, and mutation testing to effectively mitigate theinterference caused by identifiers, syntax structures, and operators on evaluation results. Experimental resultsdemonstrate that in the tasks of code generation and migration in Java and Python, CodeScore-R outperforms otherevaluation metrics and is more closely aligned with the Pass@k metric, while exhibiting stronger robustness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。