一种用于弥合软件工程任务中人类评估差距的LLM作为评审员指标
An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
AI总结：
提出SE-Jury，一种基于LLM集成评审的评估指标，通过动态选择评审团队评估生成软件工件的正确性，在三个SE任务中与人类判断的相关性提升29.6%-140.8%，接近人工一致性。
摘要：
大型语言模型（LLMs）及其他自动化技术已被越来越多地用于通过生成软件工件（如代码片段、补丁和注释）来支持软件开发人员。然而，准确评估这些生成工件的正确性仍然是一个重大挑战。一方面，人工评估提供了高准确性，但劳动密集且缺乏可扩展性。另一方面，许多自动评估指标具有可扩展性且需要最少的人力，但它们往往无法准确反映所生成软件工件的实际正确性。\n在本文中，我们提出了SE-Jury，这是第一个专门设计用于准确评估生成软件工件正确性的LLM作为集成评审员（LLM-as-Ensemble-Judge）评估指标。SE-Jury首先定义了五种不同的评估策略，每种策略由独立的评审员实施。然后，一个动态团队选择机制识别最合适的评审员子集作为一个团队，通过集成产生最终的正确性评分。我们在涵盖三个流行软件工程（SE）任务（代码生成、自动程序修复和代码摘要）的多样化SE基准上评估了SE-Jury。结果表明，SE-Jury始终与人类判断实现更高的相关性，相较于现有自动指标，改进幅度从29.6%到140.8%不等。在代码生成和程序修复中，SE-Jury与人类标注者达到的一致性水平接近标注者间一致性。这些发现强调了SE-Jury作为这些SE任务中可扩展且可靠的人工评估替代方案的潜力。
英文摘要：
Large Language Models (LLMs) and other automated techniques have been increasingly used to support software developers by generating software artifacts such as code snippets, patches, and comments. However, accurately assessing the correctness of these generated artifacts remains a significant challenge. On one hand, human evaluation provides high accuracy but is labor-intensive and lacks scalability. On the other hand, many automatic evaluation metrics are scalable and require minimal human effort, but they often fail to accurately reflect the actual correctness of generated software artifacts. In this paper, we present SE-Jury, the first evaluation metric for LLM-as-Ensemble-Judge specifically designed to accurately assess the correctness of generated software artifacts. SE-Jury first defines five distinct evaluation strategies, each implemented by an independent judge. A dynamic team selection mechanism then identifies the most appropriate subset of judges as a team to produce a final correctness score through ensembling. We evaluate SE-Jury across a diverse set of software engineering (SE) benchmarks that span three popular SE tasks: code generation, automated program repair, and code summarization. Results demonstrate that SE-Jury consistently achieves a higher correlation with human judgments, with improvements ranging from 29.6% to 140.8% over existing automatic metrics. SE-Jury reaches agreement levels with human annotators that are close to inter-annotator agreement in code generation and program repair. These findings underscore SE-Jury's potential as a scalable and reliable alternative to human evaluation in these SE tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
