学生在学习编程时常常难以解决编程问题,尤其是在线上进行时,在线学习最常见的缺点之一就是缺乏个性化帮助。这种帮助可以通过下一步提示生成来提供,即向学生展示他们接下来需要采取的具体小步骤以得出正确解决方案。生成此类提示的方法有很多,其中大型语言模型(LLMs)是目前研究最活跃的方向之一。虽然 LLMs 构成了一项很有前景的个性化帮助技术,但将其与静态分析等其他技术结合,可以显著提高输出质量。在本研究中,我们利用这一思想,提出了一个新颖的系统,为编程任务提供文本和代码提示。所提方法的流程使用思维链提示技术,包含三个不同步骤:(1)生成子目标——即从当前学生解决方案出发继续完成任务的一系列行动;(2)生成实现下一个子目标的代码;(3)生成描述该所需行动的文本。在第二步中,我们对生成的代码应用静态分析以控制其大小和质量。该工具作为对开源 JetBrains Academy 插件的修改实现,支持学生进行 IDE 内课程学习。为评估我们的方法,我们为流程中的所有步骤提出了一系列标准,并进行了两轮专家验证。最后,我们在课堂环境中对来自两所大学的 14 名学生评估了下一步提示。结果表明,文本和代码两种形式的提示对学生都有帮助,所提出的系统帮助他们推进了编程任务。
英文摘要:
Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working online being the lack of personalized help. This help can be provided as next-step hint generation, i.e., showing a student what specific small step they need to do next to get to the correct solution. There are many ways to generate such hints, with large language models (LLMs) being among the most actively studied right now.
While LLMs constitute a promising technology for providing personalized help, combining them with other techniques, such as static analysis, can significantly improve the output quality. In this work, we utilize this idea and propose a novel system to provide both textual and code hints for programming tasks. The pipeline of the proposed approach uses a chain-of-thought prompting technique and consists of three distinct steps: (1) generating subgoals - a list of actions to proceed with the task from the current student's solution, (2) generating the code to achieve the next subgoal, and (3) generating the text to describe this needed action. During the second step, we apply static analysis to the generated code to control its size and quality. The tool is implemented as a modification to the open-source JetBrains Academy plugin, supporting students in their in-IDE courses.
To evaluate our approach, we propose a list of criteria for all steps in our pipeline and conduct two rounds of expert validation. Finally, we evaluate the next-step hints in a classroom with 14 students from two universities. Our results show that both forms of the hints - textual and code - were helpful for the students, and the proposed system helped them to proceed with the coding tasks.
Large Language Models (GPT) for automating feedback on programming assignments
Institute of Information Technology, Warsaw University of Life Sciences(华沙生命科学大学信息技术研究所);Penn Center for Learning Analytics, University of Pennsylvania(宾夕法尼亚大学宾夕法尼亚学习分析中心)
astra-langchain4j: Experiences Combining LLMs and Agent Programming
School of Computer Science, University College Dublin, Ireland(都柏林大学计算机科学学院);School of Computer Science, University of St.Gallen, St. Gallen, Switzerland(圣加尔登大学计算机科学学院)
Hint of Pseudo Code (HoPC): Zero-Shot Step by Step Pseudo Code Reasoning Prompting
Department of Computer Science and Technology(计算机科学与技术系);Institute for Artificial Intelligence at Tsinghua University(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology(北京信息科学国家研究中心);Tsinghua University(清华大学)
Design and Deployment of a Course-Aware AI Tutor in an Introductory Programming Course
Johannes Kepler University Linz, Institute of Business Informatics – Software Engineering(林茨约翰内斯·开普勒大学商业信息学研究所-软件工程);University of Innsbruck, Department of Computer Science(因斯布鲁克大学生物科学系)