arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.07599cs.SEcs.AI
收藏

测试不同提示词具体程度下的大语言模型代码生成能力

Testing LLMs on Code Generation with Varying Levels of Prompt Specificity

Vanderbilt University(范德堡大学)

Lincoln Murr, Morgan Grainger, David Gao

AI总结:

本研究评估了Bard、ChatGPT-3.5、ChatGPT-4和Claude-2在104个编程问题上生成Python代码的性能,发现提示词具体程度显著影响代码准确性、时间和空间效率,并揭示了最佳提示策略。

摘要:

大语言模型(LLMs)在模仿人类文本生成和处理方面展现了无与伦比的能力。在众多受益于大语言模型的应用中,自动化代码生成正变得越来越有前景。将自然语言提示词转化为可执行代码的潜力,预示着软件开发实践的重大转变,并为显著减少手动编码工作和人为错误可能性铺平了道路。本文报告了一项研究的结果,该研究评估了多种大语言模型(如Bard、ChatGPT-3.5、ChatGPT-4和Claude-2)在生成解决编程问题的Python代码方面的性能。我们重点关注提示词的具体程度如何影响生成代码的准确性、时间效率和空间效率。我们采用了一个包含104个编程问题的基准,每个问题有四种具有不同测试程度和具体程度的提示词类型,以全面考察这些方面。我们的结果表明,不同大语言模型和提示词类型之间的性能存在显著差异,其关键贡献在于揭示了创建准确Python函数的最佳提示策略。这项研究为进一步研究大语言模型的能力奠定了基础,并为在自动化代码生成任务和测试驱动开发中利用大语言模型提出了实际意义。

英文摘要:

Large language models (LLMs) have demonstrated unparalleled prowess in mimicking human-like text generation and processing. Among the myriad of applications that benefit from LLMs, automated code generation is increasingly promising. The potential to transform natural language prompts into executable code promises a major shift in software development practices and paves the way for significant reductions in manual coding efforts and the likelihood of human-induced errors. This paper reports the results of a study that evaluates the performance of various LLMs, such as Bard, ChatGPT-3.5, ChatGPT-4, and Claude-2, in generating Python for coding problems. We focus on how levels of prompt specificity impact the accuracy, time efficiency, and space efficiency of the generated code. A benchmark of 104 coding problems, each with four types of prompts with varying degrees of tests and specificity, was employed to examine these aspects comprehensively. Our results indicate significant variations in performance across different LLMs and prompt types, and its key contribution is to reveal the ideal prompting strategy for creating accurate Python functions. This study lays the groundwork for further research in LLM capabilities and suggests practical implications for utilizing LLMs in automated code generation tasks and test-driven development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑