arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.18151cs.CYcs.HC
收藏

Howzat? 诉诸专家判断以评估面向新手程序员的人类与AI下一步提示

Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers

Neil C. C. Brown, Pierre Weill-Tessier, Juho Leinonen, Paul Denny, Michael Kölling

AI总结:

本研究通过44名Java教育工作者对LLMs和人类专家生成的编程下一步提示进行盲评排名,发现使用多阶段提示的GPT-4在生成教学提示上优于人类专家,且提示质量主要受长度和阅读水平影响,建议将提示任务嵌入专家设计的工具中。

摘要:

动机:学习编程的学生常常陷入停滞状态,无法取得进展。自动生成的下一步提示能帮助他们取得进展并支持其学习。了解什么是好提示或坏提示,以及如何在新手编程工具中(例如使用大型语言模型(LLMs))自动生成好提示非常重要。方法与参与者:我们从全球招募了44名Java教育工作者参与在线研究。我们使用一组真实的学生代码状态作为提示生成场景。参与者采用一种名为比较判断的技术,对一组候选的Java下一步提示进行排名,这些提示由大型语言模型(LLMs)和五名经验丰富的人类教育工作者生成。参与者在不知道提示如何生成的情况下对其进行排名。发现:我们发现LLMs在为编程新手生成高质量下一步提示方面存在相当大的差异,其中GPT-4的表现优于其他测试模型。当与精心设计的提示一起使用时,GPT-4在生成具有教学价值的提示方面优于人类专家。多阶段提示是最有效的LLM提示。我们发现好提示的两个最重要因素是长度(80-160词最佳)和阅读水平(美国9年级或以下最佳)。提供解决问题的替代方法被认为是糟糕的,且未发现情感倾向有任何影响。结论:鉴于LLMs的表现优于人类——即使在学生任务未知的情况下——自动生成这些提示是立即可行的。只有最佳提示才能实现这一结果,这意味着学生自己不太可能产生相同的益处。因此,提示任务应嵌入到专家设计的工具中。

英文摘要:

Motivation: Students learning to program often reach states where they are stuck and can make no forward progress. An automatically generated next-step hint can help them make forward progress and support their learning. It is important to know what makes a good hint or a bad hint, and how to generate good hints automatically in novice programming tools, for example using Large Language Models (LLMs). Method and participants: We recruited 44 Java educators from around the world to participate in an online study. We used a set of real student code states as hint-generation scenarios. Participants used a technique known as comparative judgement to rank a set of candidate next-step Java hints, which were generated by Large Language Models (LLMs) and by five human experienced educators. Participants ranked the hints without being told how they were generated. Findings: We found that LLMs had considerable variation in generating high quality next-step hints for programming novices, with GPT-4 outperforming other models tested. When used with a well-designed prompt, GPT-4 outperformed human experts in generating pedagogically valuable hints. A multi-stage prompt was the most effective LLM prompt. We found that the two most important factors of a good hint were length (80--160 words being best), and reading level (US grade 9 or below being best). Offering alternative approaches to solving the problem was considered bad, and we found no effect of sentiment. Conclusions: Automatic generation of these hints is immediately viable, given that LLMs outperformed humans -- even when the students' task is unknown. The fact that only the best prompts achieve this outcome suggests that students on their own are unlikely to be able to produce the same benefit. The prompting task, therefore, should be embedded in an expert-designed tool.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑