arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01754cs.SEcs.AI
收藏

Peer-aided Repairer:赋能大语言模型修复高级学生作业

Peer-aided Repairer: Empowering Large Language Models to Repair Advanced Student Assignments

Beihang University(北京航空航天大学);Peking University(北京大学)

Qianhui Zhao, Fang Liu, Li Zhang, Yang Liu, Zhen Yan, Zhenghao Chen, Yufei Zhou, Jing Jiang, Ge Li

AI总结:

本文针对高级编程作业修复难题,提出基于大语言模型的PaR框架,通过同伴方案选择、多源提示生成与程序修复三阶段,在Defects4DS和ITSP数据集上实现修复率新突破。

摘要:

编程作业反馈的自动生成对编程教育具有显著益处,尤其是在高级作业方面。自动程序修复技术,特别是基于大语言模型的方法,因其修复入门级作业的潜力而获得了广泛关注。然而,用于评估的程序相对简单。现有方法在修复更高层次编程课程中的程序时表现如何,仍不清楚。为了解决这些局限性,我们从一个更高层次的编程课程中整理了一个名为Defects4DS的新高级学生作业数据集。随后,我们识别了与修复高级作业中缺陷相关的挑战。基于分析,我们开发了一个由大语言模型驱动的框架,称为PaR。PaR分三个阶段工作:同伴解决方案选择、多源提示生成和程序修复。同伴解决方案选择基于词法、语义和句法标准识别密切相关的同伴程序。然后,多源提示生成巧妙地结合多个信息源,为最后的程序修复阶段创建全面且信息丰富的提示。在Defects4DS和另一个经过充分研究的ITSP数据集上的评估表明,PaR实现了新的最先进性能,与先前最先进的基于大语言模型和基于符号的方法相比,修复率分别显著提高了19.94%和15.2%。

英文摘要:

Automated generation of feedback on programming assignments holds significant benefits for programming education, especially when it comes to advanced assignments. Automated Program Repair techniques, especially Large Language Model based approaches, have gained notable recognition for their potential to fix introductory assignments. However, the programs used for evaluation are relatively simple. It remains unclear how existing approaches perform in repairing programs from higher-level programming courses. To address these limitations, we curate a new advanced student assignment dataset named Defects4DS from a higher-level programming course. Subsequently, we identify the challenges related to fixing bugs in advanced assignments. Based on the analysis, we develop a framework called PaR that is powered by the LLM. PaR works in three phases: Peer Solution Selection, Multi-Source Prompt Generation, and Program Repair. Peer Solution Selection identifies the closely related peer programs based on lexical, semantic, and syntactic criteria. Then Multi-Source Prompt Generation adeptly combines multiple sources of information to create a comprehensive and informative prompt for the last Program Repair stage. The evaluation on Defects4DS and another well-investigated ITSP dataset reveals that PaR achieves a new state-of-the-art performance, demonstrating impressive improvements of 19.94% and 15.2% in repair rate compared to prior state-of-the-art LLM- and symbolic-based approaches, respectively

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑