ChatGPT-3.5在问答与代码维护方面的实证研究
An empirical study of ChatGPT-3.5 on question answering and code maintenance
AI总结:
本研究通过问答质量对比、开发者偏好调研、代码维护任务测试三类实验,系统评估ChatGPT在技术问答与软件维护场景下的能力,为行业应用提供参考。
摘要:
自2022年ChatGPT推出以来,人们越来越担忧ChatGPT是否会取代程序员、削减工作岗位。受这一普遍担忧的驱动,我们开展了一项实证研究,系统对比ChatGPT与程序员在问答和软件维护任务中的表现。我们复用了既往研究提出的数据集,该数据集包含357个GitHub项目的Java开发者引用的130条StackOverflow(SO)讨论线程。我们主要围绕三个研究问题(RQ)展开:第一,在回答技术问题时,ChatGPT与程序员相比表现如何?第二,开发者如何看待ChatGPT的回答与SO回答之间的差异?第三,在针对维护需求修改代码时,ChatGPT与人类相比表现如何?\n针对研究问题1,我们将130个SO问题提交给ChatGPT,从相关性、可读性、信息含量、全面性和可复用性五个维度,人工对比ChatGPT的回答与SO上被采纳/最受欢迎的回答。针对研究问题2,我们邀请30名开发者开展用户研究,要求每名开发者在不知道信息来源(即ChatGPT或SO)的情况下,评估并对比10组回答。针对研究问题3,我们从引用了所研究SO线程的48个GitHub项目中提取出48项软件维护任务,要求ChatGPT修改给定的Java文件,为指定的维护需求实现对应代码。我们的研究发现了值得关注的现象:在绝大多数SO问题(130个中的97个)上,ChatGPT给出的回答更优;在300次评分中,有203次开发者更偏好ChatGPT的回答而非SO回答;在48项任务中,ChatGPT正确完成了22项的代码修改。本研究将拓展人们对ChatGPT能力的认知,为软件行业未来应用ChatGPT提供参考。
英文摘要:
Ever since the launch of ChatGPT in 2022, a rising concern is whether ChatGPT will replace programmers and kill jobs. Motivated by this widespread concern, we conducted an empirical study to systematically compare ChatGPT against programmers in question-answering and software-maintaining. We reused a dataset introduced by prior work, which includes 130 StackOverflow (SO) discussion threads referred to by the Java developers of 357 GitHub projects. We mainly investigated three research questions (RQs). First, how does ChatGPT compare with programmers when answering technical questions? Second, how do developers perceive the differences between ChatGPT's answers and SO answers? Third, how does ChatGPT compare with humans when revising code for maintenance requests? For RQ1, we provided the 130 SO questions to ChatGPT, and manually compared ChatGPT answers with the accepted/most popular SO answers in terms of relevance, readability, informativeness, comprehensiveness, and reusability. For RQ2, we conducted a user study with 30 developers, asking each developer to assess and compare 10 pairs of answers, without knowing the information source (i.e., ChatGPT or SO). For RQ3, we distilled 48 software maintenance tasks from 48 GitHub projects citing the studied SO threads. We queried ChatGPT to revise a given Java file, and to incorporate the code implementation for any prescribed maintenance requirement. Our study reveals interesting phenomena: For the majority of SO questions (97/130), ChatGPT provided better answers; in 203 of 300 ratings, developers preferred ChatGPT answers to SO answers; ChatGPT revised code correctly for 22 of the 48 tasks. Our research will expand people's knowledge of ChatGPT capabilities, and shed light on future adoption of ChatGPT by the software industry.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。