BEA 2025 关于 AI 驱动辅导系统的教学能力评估共享任务的发现
Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors
AI总结：
该论文概述了 BEA 2025 共享任务，旨在评估 LLM 驱动 AI 辅导系统的教学能力，涵盖错误识别、定位、指导、反馈可操作性及身份检测五个赛道，结果显示该领域仍有显著改进空间。
摘要：
该共享任务旨在评估由大型语言模型（LLMs）驱动的 AI 辅导系统的教学能力，重点评估在教育对话中旨在纠正学生错误的辅导系统回复质量。该任务包含五个赛道，旨在基于定义良好且有效的辅导回复的学习科学原则，自动评估 AI 辅导系统在错误识别、错误精确定位、提供指导和反馈可操作性等关键维度上的表现，以及专注于检测辅导系统身份的赛道。该任务吸引了各赛道超过 50 支国际团队参与。提交的模型针对黄金标准的人工标注进行了评估，结果虽然充满希望，但表明该领域仍有很大的改进空间：四个教学能力评估赛道的最佳结果在三类问题上的 macro F1 分数介于 58.34（提供指导）和 71.81（错误识别）之间，辅导系统识别赛道在 9 类任务上的最佳 F1 分数达到 96.98。在本文中，我们概述了该共享任务的主要发现，讨论了各团队采用的方法，并分析了它们的性能。与此任务相关的所有资源均已公开，以支持这一关键领域的未来研究。
英文摘要：
This shared task has aimed to assess pedagogical abilities of AI tutors powered by large language models (LLMs), focusing on evaluating the quality of tutor responses aimed at student's mistake remediation within educational dialogues. The task consisted of five tracks designed to automatically evaluate the AI tutor's performance across key dimensions of mistake identification, precise location of the mistake, providing guidance, and feedback actionability, grounded in learning science principles that define good and effective tutor responses, as well as the track focusing on detection of the tutor identity. The task attracted over 50 international teams across all tracks. The submitted models were evaluated against gold-standard human annotations, and the results, while promising, show that there is still significant room for improvement in this domain: the best results for the four pedagogical ability assessment tracks range between macro F1 scores of 58.34 (for providing guidance) and 71.81 (for mistake identification) on three-class problems, with the best F1 score in the tutor identification track reaching 96.98 on a 9-class task. In this paper, we overview the main findings of the shared task, discuss the approaches taken by the teams, and analyze their performance. All resources associated with this task are made publicly available to support future research in this critical domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
