理解语言模型生成代码中的缺陷
Understanding Defects in Generated Codes by Language Models
AI总结：
本研究分析了LLM生成代码中的367个缺陷，发现功能与算法错误占主导，并验证了五种提示工程技术能显著减少缺陷、提升代码可靠性。
摘要：
本研究调查了大型语言模型（LLMs）代码生成的可靠性，重点在于识别和分析生成代码中的缺陷。尽管LLMs在自动化代码生成方面具备先进能力，但确保输出的准确性和功能性仍然是一项重大挑战。通过使用结构化的缺陷分类方法来理解这些缺陷的本质和来源，本研究对LLMs生成的代码片段中识别出的367个缺陷进行了分类和分析，其中很大一部分是功能性和算法错误。这些错误类别指明了LLMs经常失败的关键领域，凸显了针对性改进的必要性。为了提高代码生成的准确性，本文实施了五种提示工程技术，包括Scratchpad Prompting、Program of Thoughts Prompting、Chain-of-Thought Prompting、Chain of Code Prompting和Structured Chain-of-Thought Prompting。这些技术被应用于优化输入提示，旨在减少歧义并提高模型的准确率。研究结果表明，精确且结构化的提示能够显著减少常见缺陷，从而提高LLM生成代码的可靠性。
英文摘要：
This study investigates the reliability of code generation by Large Language Models (LLMs), focusing on identifying and analyzing defects in the generated code. Despite the advanced capabilities of LLMs in automating code generation, ensuring the accuracy and functionality of the output remains a significant challenge. By using a structured defect classification method to understand their nature and origins this study categorizes and analyzes 367 identified defects from code snippets generated by LLMs, with a significant proportion being functionality and algorithm errors. These error categories indicate key areas where LLMs frequently fail, underscoring the need for targeted improvements. To enhance the accuracy of code generation, this paper implemented five prompt engineering techniques, including Scratchpad Prompting, Program of Thoughts Prompting, Chain-of-Thought Prompting, Chain of Code Prompting, and Structured Chain-of-Thought Prompting. These techniques were applied to refine the input prompts, aiming to reduce ambiguities and improve the models' accuracy rate. The research findings suggest that precise and structured prompting significantly mitigates common defects, thereby increasing the reliability of LLM-generated code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
