大型语言模型生成的代码出了什么问题？一项广泛的研究
What's Wrong with Your Code Generated by Large Language Models? An Extensive Study
AI总结：
本研究广泛评估了多个LLM在代码生成基准上的表现，发现其面对复杂问题生成代码成功率低且代码更短更复杂，并提出了错误分类法和无需训练的自我批评迭代方法以提升代码质量。
摘要：
大型语言模型（LLM）在代码生成方面的日益发展引起了研究人员的广泛关注。为了增强基于LLM的代码生成能力，当前的努力主要集中于收集高质量数据集和利用多样化的训练技术。然而，现有方法在局限性和边界方面的全面研究明显缺乏。为弥补这一空白，我们进行了一项广泛的实证研究，评估了三个领先的闭源LLM和六个流行的开源LLM在三个常用基准上的性能。我们的调查评估了生成代码的长度、圈复杂度和API数量，揭示了这些LLM在生成更复杂问题的成功代码方面面临挑战，并且倾向于生成比规范解决方案更短但更复杂的代码。此外，我们为错误代码开发了一个包含三个类别和十个子类别的错误分类法，并分析了常见错误类型的根本原因。为了更好地理解LLM在真实项目中的表现，我们还手动创建了一个真实世界基准RWPB。我们分析了RWPB上的错误，以突出实际场景与现有基准之间错误分布的显著差异。最后，我们提出了一种新颖的无需训练的迭代方法，引入自我批评，使LLM能够根据错误类型和编译器反馈来批评和纠正其生成的代码。我们全面而广泛的研究提供了对基于LLM的代码生成当前局限性的见解，以及提高生成代码准确性和质量的机会。
英文摘要：
The increasing development of LLMs in code generation has drawn significant attention among researchers. To enhance LLM-based code generation ability, current efforts are predominantly directed towards collecting high-quality datasets and leveraging diverse training technologies. However, there is a notable lack of comprehensive studies examining the limitations and boundaries of existing methods. To bridge this gap, we conducted an extensive empirical study evaluating the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks. Our investigation, which evaluated the length, cyclomatic complexity and API number of the generated code, revealed that these LLMs face challenges in generating successful code for more complex problems, and tend to produce code that is shorter yet more complicated as compared to canonical solutions. Additionally, we developed a taxonomy of bugs for incorrect codes that includes three categories and ten sub-categories, and analyzed the root cause for common bug types. To better understand the performance of LLMs in real-world projects, we also manually created a real-world benchmark RWPB. We analyzed bugs on RWPB to highlight distinct differences in bug distributions between actual scenarios and existing benchmarks. Finally, we propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback. Our comprehensive and extensive study provides insights into the current limitations of LLM-based code generation and opportunities for enhancing the accuracy and quality of the generated code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
