利用大型语言模型判断评估与优化教育内容
Evaluating and Optimizing Educational Content with Large Language Model Judgments
AI总结：
本文提出用语言模型作为教育专家评估教学效果，并以语言模型判断为奖励函数优化生成数学练习题，验证了其与人类教师偏好的一致性。
摘要：
创建有效的教育材料通常需要对学生的学习成果进行昂贵且耗时的研究。为了克服这一障碍，一种思路是构建学生学习的计算模型，并利用这些模型来优化教学材料。然而，对学习动态的认知过程进行建模十分困难。我们提出一种替代方法，将语言模型（LMs）作为教育专家，用于评估不同教学方式对学习成果的影响。具体而言，我们使用 GPT-3.5 来评估教学材料对不同学生群体的整体效果，并发现它能够复现诸如专长反转效应（Expertise Reversal Effect）和变异性效应（Variability Effect）等已被充分验证的教育研究发现。这表明语言模型有潜力成为可靠的教育内容评估者。基于这一认识，我们引入一种指令优化方法：由一个语言模型生成教学材料，并以另一个语言模型的判断作为奖励函数。我们将该方法应用于生成旨在最大化学生学习收益的数学应用题练习册。人类教师对这些由语言模型生成的练习册的评估显示，语言模型的判断与人类教师的偏好之间存在显著的一致性。最后，我们讨论了人类与语言模型意见之间可能存在的分歧，以及自动化教学设计由此带来的潜在隐患。
英文摘要：
Creating effective educational materials generally requires expensive and time-consuming studies of student learning outcomes. To overcome this barrier, one idea is to build computational models of student learning and use them to optimize instructional materials. However, it is difficult to model the cognitive processes of learning dynamics. We propose an alternative approach that uses Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. Specifically, we use GPT-3.5 to evaluate the overall effect of instructional materials on different student groups and find that it can replicate well-established educational findings such as the Expertise Reversal Effect and the Variability Effect. This demonstrates the potential of LMs as reliable evaluators of educational content. Building on this insight, we introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. We apply this approach to create math word problem worksheets aimed at maximizing student learning gains. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences. We conclude by discussing potential divergences between human and LM opinions and the resulting pitfalls of automating instructional design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
