TAMIGO：在高级计算课程中利用LLM辅助口试与代码评估以赋能助教
TAMIGO: Empowering Teaching Assistants using LLM-assisted viva and code assessment in an Advanced Computing Class
AI总结：
本文提出TAMIGO系统，利用LLM辅助助教进行口试与代码评估，发现其在生成口试问题时非常有效，反馈全面且具建设性，但存在幻觉问题且与评分标准对齐仍有改进空间。
摘要：
大型语言模型（LLMs）显著改变了教育领域，为学生、讲师和助教提供了新工具。本文研究LLMs在印度某大学分布式系统高级计算课程中，辅助助教进行口试和代码评估的应用。我们开发了TAMIGO，一个供助教评估编程作业的基于LLM的系统。对于口试评估，助教使用TAMIGO生成问题并将其分发给学生作答。随后，助教使用TAMIGO对学生答案生成反馈。对于代码评估，助教从学生代码提交中选择特定代码块并输入TAMIGO，以生成针对这些代码块的反馈。助教利用TAMIGO生成的关于学生答案和代码块的反馈进行进一步评估。我们评估了LLM生成的口试问题、标准答案、口试答案反馈以及学生代码提交反馈的质量。结果表明，当提供充足的上下文和背景信息时，LLM在生成口试问题方面非常有效。然而，LLM生成的口试答案反馈结果喜忧参半；幻觉现象偶尔会降低反馈的准确性。尽管如此，该反馈仍然是一致的、建设性的、全面的、平衡的，且不会让助教感到应接不暇。类似地，对于代码提交，LLM生成的反馈具有建设性、全面性和平衡性，尽管在使反馈与讲师提供的代码评估评分标准保持一致方面仍有改进空间。我们的发现有助于理解将LLM整合到教育环境中的益处和局限性。
英文摘要：
Large Language Models (LLMs) have significantly transformed the educational landscape, offering new tools for students, instructors, and teaching assistants. This paper investigates the application of LLMs in assisting teaching assistants (TAs) with viva and code assessments in an advanced computing class on distributed systems in an Indian University. We develop TAMIGO, an LLM-based system for TAs to evaluate programming assignments. For viva assessment, the TAs generated questions using TAMIGO and circulated these questions to the students for answering. The TAs then used TAMIGO to generate feedback on student answers. For code assessment, the TAs selected specific code blocks from student code submissions and fed it to TAMIGO to generate feedback for these code blocks. The TAMIGO-generated feedback for student answers and code blocks was used by the TAs for further evaluation. We evaluate the quality of LLM-generated viva questions, model answers, feedback on viva answers, and feedback on student code submissions. Our results indicate that LLMs are highly effective at generating viva questions when provided with sufficient context and background information. However, the results for LLM-generated feedback on viva answers were mixed; instances of hallucination occasionally reduced the accuracy of feedback. Despite this, the feedback was consistent, constructive, comprehensive, balanced, and did not overwhelm the TAs. Similarly, for code submissions, the LLM-generated feedback was constructive, comprehensive and balanced, though there was room for improvement in aligning the feedback with the instructor-provided rubric for code evaluation. Our findings contribute to understanding the benefits and limitations of integrating LLMs into educational settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
