评估ChatGPT在自动化代码生成领域的前景与陷阱
Assessing the Promise and Pitfalls of ChatGPT for Automated Code Generation
AI总结：
本研究构建含131个提示词的数据集，通过14项代码质量指标对比评估ChatGPT与人类的代码生成能力，明确其优势与局限，验证了AI与人类代码风格的可区分性，为AI编程助手发展及相关研究提供支撑。
摘要：
本文对主流大语言模型ChatGPT的代码生成能力与人类程序员的表现进行了全面对比评估。研究构建了一个涵盖5大类、共131个代码生成提示词的全新数据集，以支撑可靠的分析。针对所有提示词，分别由ChatGPT和人类生成代码解决方案，最终得到262份代码样本。研究采用严谨的人工评估方法，依托14项已有的代码质量指标，重点评估正确性、可理解性与安全性。核心发现显示，ChatGPT擅长运用高级结构编写简洁高效的代码，在数据分析任务中表现突出（准确率达93.1%），但在可视化图形类难题中存在局限。与人类代码的对比分析表明，ChatGPT更倾向于模块化设计，且错误处理能力更优。此外，机器学习模型可有效区分ChatGPT与人类编写的代码，准确率最高达88%，说明二者的编码风格存在可检测的差异。本研究通过量化指标与定性分析，深入揭示了ChatGPT的代码生成能力与局限性，为推动AI编程助手的发展提供了有价值的参考。所构建的数据集与评估方法也为这一新兴领域的后续研究奠定了坚实基础。所有数据与代码可访问https://github.com/DSAatUSU/ChatGPT-promises-and-pitfalls获取。
英文摘要：
This paper presents a comprehensive evaluation of the code generation capabilities of ChatGPT, a prominent large language model, compared to human programmers. A novel dataset of 131 code-generation prompts across 5 categories was curated to enable robust analysis. Code solutions were generated by both ChatGPT and humans for all prompts, resulting in 262 code samples. A meticulous manual assessment methodology prioritized evaluating correctness, comprehensibility, and security using 14 established code quality metrics. The key findings reveal ChatGPT's strengths in crafting concise, efficient code with advanced constructs, showcasing strengths in data analysis tasks (93.1% accuracy) but limitations in visual-graphical challenges. Comparative analysis with human code highlights ChatGPT's inclination towards modular design and superior error handling. Additionally, machine learning models effectively distinguished ChatGPT from human code with up to 88% accuracy, suggesting detectable coding style disparities. By providing profound insights into ChatGPT's code generation capabilities and limitations through quantitative metrics and qualitative analysis, this study makes valuable contributions toward advancing AI-based programming assistants. The curated dataset and methodology offer a robust foundation for future research in this nascent domain. All data and codes are available on https://github.com/DSAatUSU/ChatGPT-promises-and-pitfalls.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
