发表机构
University of Colombo School of Computing; University of the Sunshine Coast（科伦坡大学计算机学院; 阳光海岸大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文通过对157名学生的调查和20名学生的准实验，研究了LLM辅助软件工程的效果，发现其益处取决于任务类型，受专业水平和验证实践调节，且会改变工作流程结构。
AI 中文摘要
大型语言模型（LLMs）正日益被集成到软件开发工作流程中，但人们在讨论其效果时往往未区分任务类型、开发者资历和验证需求。本文针对一年级和四年级本科生开展了LLM辅助软件工程的混合方法实证研究。第一阶段是初步调查（样本量n=157），刻画两组学生的LLM接触情况、依赖程度及信任校准情况。第二阶段是针对两组的目的性样本（样本量n=20）开展的基于任务的准实验，在此阶段，我们比较了AI辅助与非AI条件下，涵盖实现、约束驱动算法选择和架构推理的结构化软件工程任务的表现，随后结合通过屏幕录制和定性编码流程捕获的行为轨迹分析绩效结果。调查结果显示LLM被广泛采用且伴随大量验证工作，同时两组学生在约束密集场景下对LLM能力的感知存在差异。准实验进一步表明，AI辅助会改变工作流程结构，例如参与者常采用AI优先的任务输入、复制-转移整合和AI介导的调试，而非AI工作流程则更多依赖文档、现有模板和迭代试错优化。总体而言，我们的研究结果表明，LLM辅助的益处取决于任务类型，并受专业水平和验证实践的调节，而非仅由生成速度决定。
英文摘要
Large Language Models (LLMs) are increasingly integrated into software development workflows, yet their effects are often discussed without distinguishing between task types, developer seniority, and verification demands. This paper presents a mixed-method empirical study of LLM-assisted software engineering with first-year and fourth-year undergraduates. Phase 1 is a preliminary survey (n=157) that characterizes LLM exposure, reliance, and trust calibration among the two groups. Phase 2 is a task-based quasi-experiment with a purposive sample from both cohorts (n=20). Here, we compare AI-assisted and non-AI conditions on a structured set of software engineering tasks spanning implementation, constraint-driven algorithm selection, and architectural reasoning. We then analyze performance outcomes alongside behavioral traces captured via screen recording and a qualitative coding process. Survey results indicate widespread LLM adoption and substantial verification effort, alongside cohort differences in perceived LLM capability for constraint-heavy scenarios. The quasi-experiment further shows that AI assistance changes workflow structure. For example, participants frequently adopt AI-first task entry, copy-transfer integration, and AI-mediated debugging, whereas non-AI workflows rely more on documentation, prior templates, and iterative trial-error refinement. Overall, our findings suggest that the benefits of LLM assistance are task-dependent and mediated by expertise and verification practices, rather than by generation speed alone.
CommentsPresented at ENASE 2026 : 21st International Conference on Evaluation of Novel Approaches to Software Engineering
Journal refIn Proceedings of the 21st International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE 2026; ISBN 978-989-758-828-0; ISSN 2184-4895, SciTePress, pages 210-222
