TeachObs：多模态教学观察与模型评估的人工验证基准
TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation
AI总结：
提出TeachObs基准，包含30节公开课视频的5158个15秒场景，由7名研究者标注39个二值观察码，并评估5个前沿视觉大语言模型在三种任务上的表现。
摘要：
课堂视频包含可观察的教学实践，但其教学和视觉信号很少以适合模型评估的形式组织。我们提出了 extit{TeachObs}，一个用于课堂视频中多模态教学观察的人工验证基准。 extit{TeachObs}包含来自八个国家的30节公开课视频，分为5158个固定的15秒场景。七名研究者用39个二值观察码标注每个场景，涵盖20个视觉码（如手势、板书、指向和视觉材料）和19个非视觉码（如指导、监控、提问、反馈和反思）。基于Krippendorff's alpha，使用可靠性和流行度感知规则构建黄金片段标签。除了片段级标签，三名专家评分员对30节课进行了课程级评分和定性评估，涵盖教学设计、教学实施、学生反应、学习材料和课程收尾，评分员覆盖范围详见正文。利用这两个人工参考层，我们在三个任务上评估了五个具备视觉能力的前沿大语言模型——纯文本片段编码、文本+帧片段编码，以及在LLM作为评判协议下的课程级覆盖评分——发现没有单一模型在所有三个任务中持续优于其他模型，添加中间帧会同时增加每个场景的真实和虚假归因，并且模型评估相对于专家评分员高估了程序清晰的课程。因此， extit{TeachObs}支持细粒度标注基准测试和整课评估，展示了AI系统在哪些方面可以辅助课堂视频分析，以及在哪些方面专家判断仍然必要，涵盖不同学科、课堂形式和标注难度水平。
英文摘要：
Classroom videos contain observable teaching practices, but their pedagogical and visual signals are rarely organized in forms suitable for model evaluation. We present \textit{TeachObs}, a human-validated benchmark for multimodal teaching observation in classroom videos. \textit{TeachObs} includes 30 public lesson videos from eight countries divided into 5,158 fixed 15-second scenes. Seven researchers annotated each scene with 39 binary observation codes, covering 20 visual codes, such as gesture, board work, pointing, and visual materials, and 19 nonvisual codes, such as instruction, monitoring, questioning, feedback, and reflection. Gold segment labels are constructed using reliability- and prevalence-aware rules based on Krippendorff's alpha. In addition to segment-level labels, three expert raters produced lesson-level ratings and qualitative evaluations of instructional design, instructional delivery, learner response, learning materials, and lesson closure across the 30 lessons, with rater coverage detailed in the body. Using these two human reference layers, we evaluate five vision-capable frontier LLMs across three tracks - text-only segment coding, text + frame segment coding, and lesson-level coverage scored under an LLM-as-judge protocol - and find that no single model consistently outperforms others across all three tracks, that adding a mid-frame inflates both true and false attributions per scene, and that model evaluations over-rate procedurally clear lessons relative to expert raters. \textit{TeachObs} therefore supports both fine-grained annotation benchmarking and whole-lesson evaluation, showing where AI systems can assist classroom video analysis and where expert judgment remains necessary across varied subjects, classroom formats, and annotation difficulty levels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
