arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.18181cs.SE
收藏

关于大型语言模型在单元测试生成中的评估

On the Evaluation of Large Language Models in Unit Test Generation

Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, Junjie Chen

AI总结:

本文首次基于17个Java项目和五个开源LLMs,实证评估了不同提示策略对单元测试生成的影响,揭示了其相比GPT-4和EvoSuite的性能与局限,为未来研究和实践提供了指导。

摘要:

单元测试是软件开发中验证软件组件正确性的重要活动。然而,手动编写单元测试既具挑战性又耗时。大型语言模型(LLMs)的出现为自动化单元测试生成提供了新方向。现有研究主要关注采用固定提示策略的闭源LLMs(如ChatGPT和CodeX),而未探索具有各种提示设置的先进开源LLMs的能力。特别是,开源LLMs在数据隐私保护方面具有优势,并在某些任务中表现出色。此外,有效的提示对于最大化LLMs的能力至关重要。本文基于17个Java项目、五个具有不同结构和参数规模的广泛使用的开源LLMs以及全面的评估指标,进行了首次实证研究以填补这一空白。我们的发现强调了各种提示因素的显著影响,展示了开源LLMs与商业GPT-4和传统EvoSuite相比的性能,并识别了基于LLM的单元测试生成的局限性。随后,我们从研究中得出一系列启示,以指导未来研究和基于LLM的单元测试生成的实际应用。

英文摘要:

Unit testing is an essential activity in software development for verifying the correctness of software components. However, manually writing unit tests is challenging and time-consuming. The emergence of Large Language Models (LLMs) offers a new direction for automating unit test generation. Existing research primarily focuses on closed-source LLMs (e.g., ChatGPT and CodeX) with fixed prompting strategies, leaving the capabilities of advanced open-source LLMs with various prompting settings unexplored. Particularly, open-source LLMs offer advantages in data privacy protection and have demonstrated superior performance in some tasks. Moreover, effective prompting is crucial for maximizing LLMs' capabilities. In this paper, we conduct the first empirical study to fill this gap, based on 17 Java projects, five widely-used open-source LLMs with different structures and parameter sizes, and comprehensive evaluation metrics. Our findings highlight the significant influence of various prompt factors, show the performance of open-source LLMs compared to the commercial GPT-4 and the traditional Evosuite, and identify limitations in LLM-based unit test generation. We then derive a series of implications from our study to guide future research and practical use of LLM-based unit test generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑