解码人类与大语言模型在编程中的协作：多轮对话的真实世界经验研究
Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild
AI总结：
本研究通过实证分析探讨人类与LLM在编程中的协作机制，发现任务类型影响交互模式，LLM在调试和重构任务中表现欠佳，用户满意度受任务类型影响。
摘要：
大型语言模型（LLMs）正日益成为动态对话接口，支持多轮交互，模拟人类对话并促进复杂的任务如编程。尽管像LMSYS-Chat-1M和WildChat这样的数据集捕捉了真实世界用户与LLM的对话，但很少有研究系统地探索编程场景中的人类与LLM协作的机制。在交互过程中，用户会经历哪些曲折路径？LLMs遵循指令的能力如何？用户是否满意？本文利用LMSYS-Chat-1M和WildChat数据集对人类与LLM的编程协作进行实证分析，以探讨人类与LLM的协作机制、LLMs的指令遵循能力以及人类的满意度。本研究得出有趣的发现：1）任务类型塑造交互模式（线性、星形和树形），代码质量优化倾向于线性模式，设计驱动的任务偏向树形结构，而查询则偏好星形模式；2）调试和代码重构对LLM的指令遵循能力构成更大挑战，非合规率明显高于信息查询；3）代码质量优化和需求驱动开发任务显示出较低的用户满意度，而结构化知识查询和算法设计则显示出较高的满意度。这些见解为改进LLM界面和编程协作中的用户满意度提供了建议，同时指明了未来研究适应对话系统的方向。我们相信这项工作拓宽了对人类与LLM协同作用的理解，并支持更有效的AI辅助开发。
英文摘要：
Large language models (LLMs) are increasingly acting as dynamic conversational interfaces, supporting multi-turn interactions that mimic human-like conversation and facilitate complex tasks like coding. While datasets such as LMSYS-Chat-1M and WildChat capture real-world user-LLM conversations, few studies systematically explore the mechanisms of human-LLM collaboration in coding scenarios. What tortuous paths do users experience during the interaction process? How well do the LLMs follow instructions? Are users satisfied? In this paper, we conduct an empirical analysis on human-LLM coding collaboration using LMSYS-Chat-1M and WildChat datasets to explore the human-LLM collaboration mechanism, LLMs' instruction following ability, and human satisfaction. This study yields interesting findings: 1) Task types shape interaction patterns(linear, star and tree), with code quality optimization favoring linear patterns, design-driven tasks leaning toward tree structures, and queries preferring star patterns; 2) Bug fixing and code refactoring pose greater challenges to LLMs' instruction following, with non-compliance rates notably higher than in information querying; 3) Code quality optimization and requirements-driven development tasks show lower user satisfaction, whereas structured knowledge queries and algorithm designs yield higher levels. These insights offer recommendations for improving LLM interfaces and user satisfaction in coding collaborations, while highlighting avenues for future research on adaptive dialogue systems. We believe this work broadens understanding of human-LLM synergies and supports more effective AI-assisted development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
