arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.08430cs.SEcs.AI
收藏

分析提示词对自动化方法生成的影响:基于Copilot的实证研究

Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot

University of Milano - Bicocca(米兰比可卡大学)

Ionut Daniel Fagadau, Leonardo Mariani, Daniela Micucci, Oliviero Riganelli

AI总结:

本文系统研究八种提示词特征对Copilot生成Java代码质量的影响,发现示例和方法目的摘要等特征能显著提升生成结果的正确性、复杂度、规模和相似度。

摘要:

生成式AI正在改变开发者与软件系统交互的方式,提供能够生成并交付新内容的服务,以满足开发者的实际需求。例如,开发者可以通过编写自然语言提示词直接在IDE中请求新代码,而基于生成式AI的集成服务(如Copilot)会立即响应提示词,提供可直接使用的代码片段。恰当地编写提示词,并融入有用信息同时避免信息过载,可能是获取正确代码的关键因素。设计优质提示词的任务被称为提示工程。本文系统研究了八种提示词特征对提示词风格与内容的影响,以及对生成代码的正确性、复杂度、规模和与开发者代码相似度的影响。我们具体考虑了使用Copilot的任务,通过系统组合八种所考虑的提示词特征获得了124,800个提示词,用于生成200个Java方法的实现。结果显示,某些提示词特征(如示例的存在和方法目的的摘要)能够显著影响生成结果的质量。

英文摘要:

Generative AI is changing the way developers interact with software systems, providing services that can produce and deliver new content, crafted to satisfy the actual needs of developers. For instance, developers can ask for new code directly from within their IDEs by writing natural language prompts, and integrated services based on generative AI, such as Copilot, immediately respond to prompts by providing ready-to-use code snippets. Formulating the prompt appropriately, and incorporating the useful information while avoiding any information overload, can be an important factor in obtaining the right piece of code. The task of designing good prompts is known as prompt engineering. In this paper, we systematically investigate the influence of eight prompt features on the style and the content of prompts, on the level of correctness, complexity, size, and similarity to the developers' code of the generated code. We specifically consider the task of using Copilot with 124,800 prompts obtained by systematically combining the eight considered prompt features to generate the implementation of 200 Java methods. Results show how some prompt features, such as the presence of examples and the summary of the purpose of the method, can significantly influence the quality of the result.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑