指导而非辅助:基于LLM的多轮规划与层级提问用于苏格拉底式代码调试
Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging
AI总结:
针对LLM直接给出答案而非引导的问题,提出TreeInstruct智能体,通过状态空间规划动态构建问题树,在多轮交互中同时处理独立与依赖错误,在多错误数据集上达到最优性能,并有效引导学生独立调试代码。
摘要:
苏格拉底式提问是一种有效的教学策略,能够鼓励批判性思维和问题解决能力。大语言模型(LLM)的对话能力在提供可扩展的、实时的学生指导方面展现出巨大潜力。然而,当前的LLM往往直接给出解决方案,使其成为低效的指导者。我们在代码调试领域通过TreeInstruct来解决这一问题,这是一种由新颖的基于状态空间的规划算法引导的指导智能体。TreeInstruct通过提出探究性问题来帮助学生独立识别和解决错误。它评估学生的概念性和语法性知识,基于学生的回答和当前知识状态动态构建问题树,在多轮交互环境中同时有效地处理独立和依赖的错误。除了使用现有的单错误调试基准外,我们还构建了一个更具挑战性的多错误数据集,包含150个编码问题、错误解决方案和错误修复——所有内容均由专家精心构建和标注。广泛评估表明,TreeInstruct在两个数据集上均达到最先进的性能,证明其作为指导者比基线方法更有效。此外,一项涉及五名不同技能水平学生的真实世界案例研究进一步证明了TreeInstruct能够以最少的轮次和高度苏格拉底式的提问引导学生高效调试代码的能力。
英文摘要:
Socratic questioning is an effective teaching strategy, encouraging critical thinking and problem-solving. The conversational capabilities of large language models (LLMs) show great potential for providing scalable, real-time student guidance. However, current LLMs often give away solutions directly, making them ineffective instructors. We tackle this issue in the code debugging domain with TreeInstruct, an Instructor agent guided by a novel state space-based planning algorithm. TreeInstruct asks probing questions to help students independently identify and resolve errors. It estimates a student's conceptual and syntactical knowledge to dynamically construct a question tree based on their responses and current knowledge state, effectively addressing both independent and dependent mistakes concurrently in a multi-turn interaction setting. In addition to using an existing single-bug debugging benchmark, we construct a more challenging multi-bug dataset of 150 coding problems, incorrect solutions, and bug fixes -- all carefully constructed and annotated by experts. Extensive evaluation shows TreeInstruct's state-of-the-art performance on both datasets, proving it to be a more effective instructor than baselines. Furthermore, a real-world case study with five students of varying skill levels further demonstrates TreeInstruct's ability to guide students to debug their code efficiently with minimal turns and highly Socratic questioning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。