优化ChatGPT生成的代码：表征并缓解代码质量问题
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
AI总结：
本研究系统分析ChatGPT生成的4066份Java/Python代码的质量，识别出正确性、可维护性等问题，验证其自我修复可提效超20%，为优化AI代码生成能力提供参考。
摘要：
我们系统研究了ChatGPT为2033项编程任务生成的、用Java和Python这两种主流编程语言实现的4066份代码的质量。本研究的目标有三方面：第一，分析ChatGPT在代码生成任务上的正确性，揭示影响其效果的因素，包括任务难度、编程语言、任务提出时间和程序规模；第二，识别并表征ChatGPT生成代码的潜在质量问题；第三，为如何缓解这些问题提供洞见。\n实验结果显示，在ChatGPT生成的4066份程序中，2756份被判定为正确，1082份输出错误，177份存在编译或运行时错误。此外，我们通过静态分析工具进一步分析了生成代码的其他特征，比如代码风格和可维护性，发现1930段ChatGPT生成的代码存在可维护性问题。随后，我们探究了ChatGPT的自我修复能力，以及它与静态分析工具交互修复前述错误的效果。实验表明，ChatGPT能部分解决这些问题，将代码质量提升20%以上，但仍存在局限性，也有改进空间。总体而言，本研究为认识ChatGPT的当前局限提供了宝贵洞见，并为未来提升ChatGPT这类AI模型代码生成能力的研发工作提供了路线图。
英文摘要：
We systematically study the quality of 4,066 ChatGPT-generated code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is three folds. First, we analyze the correctness of ChatGPT on code generation tasks and uncover the factors that influence its effectiveness, including task difficulty, programming language, time that tasks are introduced, and program size. Second, we identify and characterize potential issues with the quality of ChatGPT-generated code. Last, we provide insights into how these issues can be mitigated. Experiments highlight that out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong outputs, and 177 programs contain compilation or runtime errors. Additionally, we further analyze other characteristics of the generated code through static analysis tools, such as code style and maintainability, and find that 1,930 ChatGPT-generated code snippets suffer from maintainability issues. Subsequently, we investigate ChatGPT's self-repairing ability and its interaction with static analysis tools to fix the errors uncovered in the previous step. Experiments suggest that ChatGPT can partially address these challenges, improving code quality by more than 20%, but there are still limitations and opportunities for improvement. Overall, our study provides valuable insights into the current limitations of ChatGPT and offers a roadmap for future research and development efforts to enhance the code generation capabilities of AI models like ChatGPT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
