arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.20545cs.SEcs.CLcs.HCcs.LG
收藏

提示编程对函数级代码生成的影响

The Impact of Prompt Programming on Function-Level Code Generation

Chalmers University of Technology and University of Gothenburg(查尔姆斯理工大学和哥德堡大学);RISE Research Institutes of Sweden(瑞典RISE研究机构)

Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Philipp Leitner

AI总结:

本研究提出CodePromptEval数据集,评估五种提示技术对三个LLM生成函数代码的影响,发现组合技术未必提升效果,且正确性与质量存在权衡。

摘要:

大型语言模型(LLMs)正越来越多地被软件工程师用于代码生成。然而,LLMs的局限性,如生成无关或错误的代码,凸显了提示编程(或提示工程)的必要性,即工程师应用特定的提示技术(例如,思维链或输入输出示例)来改进生成的代码。虽然一些提示技术已被研究,但不同技术及其交互对代码生成的影响仍未完全理解。在本研究中,我们引入了CodePromptEval,一个包含7072个提示的数据集,旨在评估五种提示技术(少样本、角色扮演、思维链、函数签名、包列表)及其对由三个LLMs(GPT-4o、Llama3和Mistral)生成的完整函数的正确性、相似性和质量的影响。我们的研究结果表明,虽然某些提示技术显著影响生成的代码,但组合多种技术并不一定改善结果。此外,我们观察到使用提示技术时正确性与质量之间存在权衡。我们的数据集和复现包支持未来关于改进LLM生成代码和评估新提示技术的研究。

英文摘要:

Large Language Models (LLMs) are increasingly used by software engineers for code generation. However, limitations of LLMs such as irrelevant or incorrect code have highlighted the need for prompt programming (or prompt engineering) where engineers apply specific prompt techniques (e.g., chain-of-thought or input-output examples) to improve the generated code. While some prompt techniques have been studied, the impact of different techniques -- and their interactions -- on code generation is still not fully understood. In this study, we introduce CodePromptEval, a dataset of 7072 prompts designed to evaluate five prompt techniques (few-shot, persona, chain-of-thought, function signature, list of packages) and their effect on the correctness, similarity, and quality of complete functions generated by three LLMs (GPT-4o, Llama3, and Mistral). Our findings show that while certain prompt techniques significantly influence the generated code, combining multiple techniques does not necessarily improve the outcome. Additionally, we observed a trade-off between correctness and quality when using prompt techniques. Our dataset and replication package enable future research on improving LLM-generated code and evaluating new prompt techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑