LLM 作为评估者：一种评估缺陷报告摘要的新方法
LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization
AI总结：
本研究探索用 GPT-4o、LLaMA-3 和 Gemini 等 LLM 自动评估缺陷报告摘要，实验表明 LLM 表现良好且 GPT-4o 最优，同时可避免人类评估者的疲劳问题，具备规模化评估潜力。
摘要：
对软件制品进行摘要是已被深入研究的重要任务。在评估软件摘要方法时，人工判断仍是最受信任的评估方式，但这对评估者而言耗时且易产生疲劳，难以扩展和复现。大语言模型（LLMs）已在多种软件工程任务中展现出卓越能力，这促使我们探索其作为软件制品摘要方法自动评估者的潜力。本研究考察 LLM 能否有效评估缺陷报告摘要。我们开展了一项实验，将同一组缺陷摘要问题分别交给人类和三个 LLM（GPT-4o、LLaMA-3 和 Gemini）在两个任务上进行评估：从一组候选项中选出正确的缺陷报告标题和缺陷报告摘要。结果表明，LLM 在评估缺陷报告摘要方面总体表现良好，其中 GPT-4o 优于其他 LLM。此外，人类和 LLM 都表现出一致的决策行为，但人类会经历疲劳，从而随时间影响其准确率。我们的结果表明，LLM 展现出被视作缺陷报告摘要自动评估者的潜力，这可以在减少人类评估者工作量和疲劳的同时扩大评估规模。
英文摘要：
Summarizing software artifacts is an important task that has been thoroughly researched. For evaluating software summarization approaches, human judgment is still the most trusted evaluation. However, it is time-consuming and fatiguing for evaluators, making it challenging to scale and reproduce. Large Language Models (LLMs) have demonstrated remarkable capabilities in various software engineering tasks, motivating us to explore their potential as automatic evaluators for approaches that aim to summarize software artifacts. In this study, we investigate whether LLMs can evaluate bug report summarization effectively. We conducted an experiment in which we presented the same set of bug summarization problems to humans and three LLMs (GPT-4o, LLaMA-3, and Gemini) for evaluation on two tasks: selecting the correct bug report title and bug report summary from a set of options. Our results show that LLMs performed generally well in evaluating bug report summaries, with GPT-4o outperforming the other LLMs. Additionally, both humans and LLMs showed consistent decision-making, but humans experienced fatigue, impacting their accuracy over time. Our results indicate that LLMs demonstrate potential for being considered as automated evaluators for bug report summarization, which could allow scaling up evaluations while reducing human evaluators effort and fatigue.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
