arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.01329cs.SEcs.AIcs.CL
收藏

提示炼金术士:面向测试用例生成的自动化LLM定制提示优化

The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation

The Chinese University of Hong Kong(香港中文大学);Harbin Institute of Technology(哈尔滨工业大学);Monash University Malaysia(莫纳什大学马来西亚分校);Peng Cheng Laboratory(鹏城实验室)

Shuzheng Gao, Chaozheng Wang, Cuiyun Gao, Xiaoqian Jiao, Chun Yong Chong, Shan Gao, Michael Lyu

AI总结:

本文提出面向测试用例生成的自动化LLM定制提示优化方法,解决现有提示依赖人工编写且未针对不同LLM定制的问题,通过引入引导式优化和领域上下文知识提升提示质量与测试用例生成效果。

摘要:

测试用例对于验证软件应用的可靠性和质量至关重要。近期研究表明,大语言模型(LLMs)能够为给定源代码生成有用的测试用例。然而,现有工作主要依赖人工编写的普通提示,这往往导致次优结果,因为LLMs的性能可能受到提示的极大影响。此外,这些方法对所有LLMs使用相同的提示,忽视了不同LLMs可能最适合不同提示这一事实。鉴于可能的提示表述形式种类繁多,自动发现每个LLM的最优提示构成了一项重大挑战。尽管自然语言处理领域存在自动化提示优化的方法,但它们难以针对测试用例生成任务产生有效的提示。首先,这些方法通过简单地组合和变异现有提示来迭代优化提示,缺乏适当的引导,导致生成的提示缺乏多样性,并倾向于在生成的测试用例中重复相同的错误。其次,这些提示通常缺乏领域上下文知识,限制了LLMs在该任务中的性能。

英文摘要:

Test cases are essential for validating the reliability and quality of software applications. Recent studies have demonstrated the capability of Large Language Models (LLMs) to generate useful test cases for given source code. However, the existing work primarily relies on human-written plain prompts, which often leads to suboptimal results since the performance of LLMs can be highly influenced by the prompts. Moreover, these approaches use the same prompt for all LLMs, overlooking the fact that different LLMs might be best suited to different prompts. Given the wide variety of possible prompt formulations, automatically discovering the optimal prompt for each LLM presents a significant challenge. Although there are methods on automated prompt optimization in the natural language processing field, they are hard to produce effective prompts for the test case generation task. First, the methods iteratively optimize prompts by simply combining and mutating existing ones without proper guidance, resulting in prompts that lack diversity and tend to repeat the same errors in the generated test cases. Second, the prompts are generally lack of domain contextual knowledge, limiting LLMs' performance in the task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑