基于大语言模型的虚拟助教的大规模真实世界评估
A Large-Scale Real-World Evaluation of LLM-Based Virtual Teaching Assistant
AI总结：
本研究开发并部署了基于大语言模型的虚拟助教于477名研究生的AI编程课程，通过三轮调查和3869对交互分析，评估其可行性与挑战，并开源系统代码。
摘要：
由大语言模型驱动的虚拟助教（VTA）通过提供即时反馈和促进多轮交互，具有增强学生学习的潜力。然而，关于其在真实课堂中的有效性和接受度的实证研究有限，使其实际影响尚不确定。在本研究中，我们开发了一个基于大语言模型的VTA，并将其部署在一门面向477名研究生的AI编程入门课程中。为了评估学生对VTA表现的看法如何随时间演变，我们在课程的不同阶段进行了三轮全面调查。此外，我们分析了3869对学生与VTA的交互对，以识别常见的问题类型和参与模式。然后，我们将这些交互与传统的学生与人类教师交互进行比较，以评估VTA在学习过程中的作用。通过大规模实证研究和交互分析，我们评估了在真实课堂中部署VTA的可行性，并识别了更广泛采用的关键挑战。最后，我们发布了VTA系统的源代码，以促进AI驱动教育的未来发展：https://github.com/sean0042/VTA。
英文摘要：
Virtual Teaching Assistants (VTAs) powered by Large Language Models (LLMs) have the potential to enhance student learning by providing instant feedback and facilitating multi-turn interactions. However, empirical studies on their effectiveness and acceptance in real-world classrooms are limited, leaving their practical impact uncertain. In this study, we develop an LLM-based VTA and deploy it in an introductory AI programming course with 477 graduate students. To assess how student perceptions of the VTA's performance evolve over time, we conduct three rounds of comprehensive surveys at different stages of the course. Additionally, we analyze 3,869 student--VTA interaction pairs to identify common question types and engagement patterns. We then compare these interactions with traditional student--human instructor interactions to evaluate the VTA's role in the learning process. Through a large-scale empirical study and interaction analysis, we assess the feasibility of deploying VTAs in real-world classrooms and identify key challenges for broader adoption. Finally, we release the source code of our VTA system, fostering future advancements in AI-driven education: \texttt{https://github.com/sean0042/VTA}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
