arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.03210cs.CLcs.CY
收藏

语言模型能否运用苏格拉底法?代码调试实验

Can Language Models Employ the Socratic Method? Experiments with Code Debugging

UNC Charlotte(北卡罗来纳大学夏洛特分校);New York University(纽约大学)

Erfan Al-Hossami, Razvan Bunescu, Justin Smith, Ryan Teehan

AI总结:

针对苏格拉底式教学智能体缺乏训练评估数据的问题,研究构建了人工多轮苏格拉底式代码调试建议数据集,用于基准测试Flan-T5、GPT-4等多款语言模型的苏格拉底式调试能力,并开源了相关资源。

摘要:

在运用苏格拉底教学法时,教师会引导学生自行解决问题,而非直接给出答案。这一策略虽能大幅提升学习效果,但通常耗时且对认知要求较高。自动化的苏格拉底式对话智能体可补充人工教学并实现规模化应用,但其发展因缺乏合适的训练与评估数据而受阻。本文中,我们引入了一个人工构建的多轮苏格拉底式建议数据集,旨在帮助新手程序员修复简单计算问题的错误解法。该数据集随后被用于基准测试多款语言模型的苏格拉底式调试能力,测试范围涵盖对基于指令的文本到文本Transformer模型Flan-T5的微调,以及规模大得多的GPT-4的零样本与思维链提示效果。代码与数据集已通过下方链接免费开放供研究使用:https://github.com/taisazero/socratic-debugging-benchmark

英文摘要:

When employing the Socratic method of teaching, instructors guide students toward solving a problem on their own rather than providing the solution directly. While this strategy can substantially improve learning outcomes, it is usually time-consuming and cognitively demanding. Automated Socratic conversational agents can augment human instruction and provide the necessary scale, however their development is hampered by the lack of suitable data for training and evaluation. In this paper, we introduce a manually created dataset of multi-turn Socratic advice that is aimed at helping a novice programmer fix buggy solutions to simple computational problems. The dataset is then used for benchmarking the Socratic debugging abilities of a number of language models, ranging from fine-tuning the instruction-based text-to-text transformer Flan-T5 to zero-shot and chain of thought prompting of the much larger GPT-4. The code and datasets are made freely available for research at the link below. https://github.com/taisazero/socratic-debugging-benchmark

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑