关于LLM在软件工程项目中的使用与认知的实证研究
An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project
AI总结：
本研究针对214名参与团队软件工程项目的学生，分析LLM生成代码、提示词及人工集成干预情况并调研认知，发现LLM在开发早期作用显著，为提升学生生产力及教育方向调整提供参考。
摘要：
大语言模型（Large Language Models, LLMs）是人工智能领域的一大飞跃，在处理人类语言相关任务方面表现出色。尽管通用大语言模型的核心定位并非代码生成，但它们在该领域已展现出可观的应用潜力。然而，大语言模型在学术软件工程项目中的实际效用尚未得到充分探究。本研究针对214名以最多6人团队形式开展项目的学生，探究了大语言模型的效用。值得注意的是，与多数明确禁止使用大语言模型的学术课程不同，本研究依托的学术课程鼓励学生将大语言模型纳入自身的开发工具链。\n本文分析了AI生成的代码、用于代码生成的提示词，以及将代码集成到代码库所需的人工干预程度。我们还开展了一项认知研究，从计算机科学专业学生的视角，深入了解他们对大语言模型的感知效用、影响因素及未来展望。研究结果表明，大语言模型可在软件开发的早期阶段发挥关键作用，尤其在生成基础代码结构、辅助语法处理和错误调试方面效果显著。这些发现为我们提供了一个框架，指导如何有效利用大语言模型作为工具来提升软件工程专业学生的生产力，同时凸显了将教育重点转向培养学生实现成功的人机协作的必要性。
英文摘要：
Large Language Models (LLMs) represent a leap in artificial intelligence, excelling in tasks using human language(s). Although the main focus of general-purpose LLMs is not code generation, they have shown promising results in the domain. However, the usefulness of LLMs in an academic software engineering project has not been fully explored yet. In this study, we explore the usefulness of LLMs for 214 students working in teams consisting of up to six members. Notably, in the academic course through which this study is conducted, students were encouraged to integrate LLMs into their development tool-chain, in contrast to most other academic courses that explicitly prohibit the use of LLMs. In this paper, we analyze the AI-generated code, prompts used for code generation, and the human intervention levels to integrate the code into the code base. We also conduct a perception study to gain insights into the perceived usefulness, influencing factors, and future outlook of LLM from a computer science student's perspective. Our findings suggest that LLMs can play a crucial role in the early stages of software development, especially in generating foundational code structures, and helping with syntax and error debugging. These insights provide us with a framework on how to effectively utilize LLMs as a tool to enhance the productivity of software engineering students, and highlight the necessity of shifting the educational focus toward preparing students for successful human-AI collaboration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
