自动化自动评分：大语言模型作为入门编程课程的测试套件生成器
Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
AI总结：
本文研究利用GPT-4根据题目描述和参考解答自动生成CS1编程题测试套件，并在26道题、超2.5万份学生提交上验证其效果，证明LLM生成的测试套件在识别有效解答和覆盖全面性上可与教师手工套件相当，还能揭示题目歧义。
摘要：
自动评分的编程作业能够为学生提供即时反馈，并显著减少教师的手动评分时间。然而，在自动评分器中为编程问题创建全面的测试用例套件可能既耗时又复杂。定义测试套件所需的工作量可能会使一些教师不愿创建更多题目，或导致测试覆盖不足，从而可能对学生解答产生误导性反馈。这些局限性可能会减少学生在学习编程时获得及时反馈这一已被充分证明的益处。在本工作中，我们评估了在更大工作流中使用大语言模型（LLMs）自动为CS1级别编程问题生成测试套件的有效性。我们将每个问题的题目描述和参考解答提供给GPT-4，以生成可供自动评分器使用的测试套件。我们使用26个问题以及学生在入门编程课程中提交的超过25,000份解答尝试来评估所提出的方法。我们将LLM生成的测试套件与教师为每个问题创建的测试套件进行了性能比较。我们的研究结果表明，LLM生成的测试套件能够正确识别大多数有效解答，并且对于大多数问题而言，其全面性至少与教师测试套件相当。此外，LLM生成的测试套件还暴露了一些问题描述中的歧义，凸显了它们在改进自动评分和教学设计方面的潜力。
英文摘要：
Automatically graded programming assignments provide instant feedback to students and significantly reduce manual grading time for instructors. However, creating comprehensive suites of test cases for programming problems within automatic graders can be time-consuming and complex. The effort needed to define test suites may deter some instructors from creating additional problems or lead to inadequate test coverage, potentially resulting in misleading feedback on student solutions. Such limitations may reduce student access to the well-documented benefits of timely feedback when learning programming. In this work, we evaluate the effectiveness of using Large Language Models (LLMs), as part of a larger workflow, to automatically generate test suites for CS1-level programming problems. Each problem's statement and reference solution are provided to GPT-4 to produce a test suite that can be used by an autograder. We evaluate our proposed approach using a sample of 26 problems, and more than 25,000 attempted solutions to those problems, submitted by students in an introductory programming course. We compare the performance of the LLM-generated test suites against the instructor-created test suites for each problem. Our findings reveal that LLM-generated test suites can correctly identify most valid solutions, and for most problems are at least as comprehensive as the instructor test suites. Additionally, the LLM-generated test suites exposed ambiguities in some problem statements, underscoring their potential to improve both autograding and instructional design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
