ChatGPT与软件测试教育:前景与风险
ChatGPT and Software Testing Education: Promises & Perils
AI总结:
本文评估ChatGPT回答软件测试课程常见问题的能力,发现其答案与解释正确率约55.6%和53.0%,并探讨其在教学中的机遇与风险。
摘要:
在过去十年中,代码的预测语言建模已被证明是一种有价值的工具,可帮助开发者实现新型自动化。最近,我们看到了基于神经transformer架构的通用“大型语言模型”的出现,这些模型在涵盖代码和自然语言的海量人类书写文本数据集上进行了训练。然而,尽管此类模型展现出强大的表征能力,但与它们交互历来受到特定任务设置的限制,从而限制了其普遍适用性。随着ChatGPT的推出,其中许多限制最近已被克服。ChatGPT是由OpenAI创建的语言模型,经过训练后可作为对话智能体运行,使其能够回答问题并响应终端用户的各种命令。诸如ChatGPT之类模型的出现已经引发了教育工作者的热烈讨论,既有人担心学生可能使用这些AI工具规避学习,也有人对它们可能开启的新型学习机会感到兴奋。然而,鉴于这些工具尚处于初期阶段,我们目前缺乏关于它们在不同教育场景中表现如何,以及它们可能对传统教学形式构成何种潜在前景(或危险)的基础知识。因此,本文考察了ChatGPT在被要求回答一门流行软件测试课程中的常见问题时表现如何。我们的研究结果表明,ChatGPT在55.6%的情况下能够提供正确或部分正确的答案,在53.0%的情况下能够提供正确或部分正确的答案解释,并且在共享问题上下文中提示该工具会带来略高的正确回答率。基于这些发现,我们讨论了学生和教师使用ChatGPT相关的潜在前景与风险。
英文摘要:
Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the advent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text spanning code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users. The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. Our findings indicate that ChatGPT can provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct responses. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。