利用LLMs评估真实对话中的辅导行为：一项可行性研究
Leveraging LLMs to Assess Tutor Moves in Real-Life Dialogues: A Feasibility Study
AI总结：
本研究利用多种大语言模型分析真实数学辅导对话，验证了AI在检测和评估辅导行为（如表扬和错误回应）上的可行性，准确率接近人工水平，并提出经济高效的提示策略以支持大规模评估。
摘要：
辅导能提高学生的学业成绩，但基于音频转录在大规模上识别和研究哪些辅导行为与学生学习最相关，仍是一个开放的研究问题。本研究探讨了使用生成式AI识别和评估真实数学辅导中特定辅导行为的可行性和可扩展性。我们分析了50份随机选取的大学生远程辅导初中生数学的转录文本。使用GPT-4、GPT-4o、GPT-4-turbo、Gemini-1.5-pro和LearnLM，我们评估了辅导教师在两项辅导技能上的应用：给予有效表扬和回应学生的数学错误。所有模型都能可靠地检测相关情境，例如，辅导教师给予学生表扬（准确率94-98%）和学生犯数学错误（准确率82-88%），并有效评估了辅导教师对辅导最佳实践的遵循程度，与人工判断高度一致（分别为83-89%和73-77%）。我们提出了一种成本效益高的提示策略，并讨论了在真实环境中使用大语言模型支持可扩展评估的实际意义。这项工作还贡献了LLM提示词，以支持AI辅助学习研究的可重复性。
英文摘要：
Tutoring improves student achievement, but identifying and studying what tutoring actions are most associated with student learning at scale based on audio transcriptions is an open research problem. This present study investigates the feasibility and scalability of using generative AI to identify and evaluate specific tutor moves in real-life math tutoring. We analyze 50 randomly selected transcripts of college-student remote tutors assisting middle school students in mathematics. Using GPT-4, GPT-4o, GPT-4-turbo, Gemini-1.5-pro, and LearnLM, we assess tutors' application of two tutor skills: delivering effective praise and responding to student math errors. All models reliably detected relevant situations, for example, tutors providing praise to students (94-98% accuracy) and a student making a math error (82-88% accuracy) and effectively evaluated the tutors' adherence to tutoring best practices, aligning closely with human judgments (83-89% and 73-77%, respectively). We propose a cost-effective prompting strategy and discuss practical implications for using large language models to support scalable assessment in authentic settings. This work further contributes LLM prompts to support reproducibility and research in AI-supported learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
