arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.09268cs.SEcs.AIcs.CYcs.HC
收藏

一步一个脚印:结合 LLMs 与静态分析为编程任务生成下一步提示

One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks

JetBrains Research(JetBrains研究院);JetBrains(JetBrains公司);Utrecht University(乌得勒支大学)

Anastasiia Birillo, Elizaveta Artser, Anna Potriasaeva, Ilya Vlasov, Katsiaryna Dzialets, Yaroslav Golubev, Igor Gerasimov, Hieke Keuning, Timofey Bryksin

AI总结:

本研究提出结合 LLMs 与静态分析的三步流程系统,为编程任务生成文本与代码形式的下一步提示,专家验证与课堂实验表明该系统能有效帮助学生推进编程任务。

摘要:

学生在学习编程时常常难以解决编程问题,尤其是在线上进行时,在线学习最常见的缺点之一就是缺乏个性化帮助。这种帮助可以通过下一步提示生成来提供,即向学生展示他们接下来需要采取的具体小步骤以得出正确解决方案。生成此类提示的方法有很多,其中大型语言模型(LLMs)是目前研究最活跃的方向之一。虽然 LLMs 构成了一项很有前景的个性化帮助技术,但将其与静态分析等其他技术结合,可以显著提高输出质量。在本研究中,我们利用这一思想,提出了一个新颖的系统,为编程任务提供文本和代码提示。所提方法的流程使用思维链提示技术,包含三个不同步骤:(1)生成子目标——即从当前学生解决方案出发继续完成任务的一系列行动;(2)生成实现下一个子目标的代码;(3)生成描述该所需行动的文本。在第二步中,我们对生成的代码应用静态分析以控制其大小和质量。该工具作为对开源 JetBrains Academy 插件的修改实现,支持学生进行 IDE 内课程学习。为评估我们的方法,我们为流程中的所有步骤提出了一系列标准,并进行了两轮专家验证。最后,我们在课堂环境中对来自两所大学的 14 名学生评估了下一步提示。结果表明,文本和代码两种形式的提示对学生都有帮助,所提出的系统帮助他们推进了编程任务。

英文摘要:

Students often struggle with solving programming problems when learning to code, especially when they have to do it online, with one of the most common disadvantages of working online being the lack of personalized help. This help can be provided as next-step hint generation, i.e., showing a student what specific small step they need to do next to get to the correct solution. There are many ways to generate such hints, with large language models (LLMs) being among the most actively studied right now. While LLMs constitute a promising technology for providing personalized help, combining them with other techniques, such as static analysis, can significantly improve the output quality. In this work, we utilize this idea and propose a novel system to provide both textual and code hints for programming tasks. The pipeline of the proposed approach uses a chain-of-thought prompting technique and consists of three distinct steps: (1) generating subgoals - a list of actions to proceed with the task from the current student's solution, (2) generating the code to achieve the next subgoal, and (3) generating the text to describe this needed action. During the second step, we apply static analysis to the generated code to control its size and quality. The tool is implemented as a modification to the open-source JetBrains Academy plugin, supporting students in their in-IDE courses. To evaluate our approach, we propose a list of criteria for all steps in our pipeline and conduct two rounds of expert validation. Finally, we evaluate the next-step hints in a classroom with 14 students from two universities. Our results show that both forms of the hints - textual and code - were helpful for the students, and the proposed system helped them to proceed with the coding tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑