动态大语言模型（LLM）对话在软件开发中的应用前景

中文摘要 AI 辅助

大型语言模型（LLM）已成为辅助开发者的重要工具，但我们仍缺乏关于如何在开发活动中有效支持开发者与LLM交互的相关知识。与基于聊天的LLM交互的质量仍高度依赖于开发者如何措辞提示词以及包含哪些信息。本研究的目标是评估对开发者与LLM的交互进行干预是否会产生影响，无论这种影响是有害还是有益。为此，我们开展了一项为期四个月的纵向研究，对象是第三学期的计算机科学学生，他们使用基于聊天的LLM完成全栈Web开发项目，分为三个条件组：（1）情境感知组接收基于意图的对话增强；（2）主动组接收后续建议和定制化建议；（3）无干预的对照组。我们的增强措施是最小化的，目的是减少混杂因素并分离处理效应。通过分析交互日志和用户调查发现，交互模式没有显著差异，表明在测量结果中，对交互进行干预没有可检测到的有害影响。此外，我们观察到主动组的满意度有上升趋势。结果表明，即使采用最小化的干预措施，开发者与LLM交互的动态引导机制也会产生可观测的效果，因此更严格的增强措施可能有潜力大幅提升开发者的满意度。

英文摘要

Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.

发表机构

• Leipzig University（莱比锡大学）
• Chemnitz University of Technology（开姆尼茨工业大学）
• ScaDS.AI Dresden/Leipzig（ScaDS.AI德累斯顿/莱比锡）

机构由 AI 辅助整理，请以论文原文为准。