ELI-Why：评估语言模型解释的教学效用
ELI-Why: Evaluating the Pedagogical Utility of Language Model Explanations
AI总结：
该研究提出包含1.34万个“为什么”问题的ELI-Why基准，通过两项人类研究和自动化评估发现，语言模型生成的教育解释在年级适配度、用户信息需求匹配度上均显著低于人类策划内容，教学效用有限。
摘要：
当前语言模型已广泛应用于教育领域，但其针对具有不同信息需求和知识背景的学习者定制回答的能力仍未得到充分研究。为此，我们提出了ELI-Why，一个包含1.34万个“为什么”问题的基准，用于评估语言模型的教学能力。随后我们开展了两项大规模人类研究，在该基准上评估语言模型生成的解释性回答（即解释）的效用，这些解释针对三个不同的教育阶段定制：小学、高中和研究生阶段。在第一项研究中，人类评分者扮演“教育者”的角色，评估模型生成的解释对不同教育阶段的适配度。我们发现，GPT-4生成的解释与目标教育背景的匹配率仅为50%，而普通人类策划的解释匹配率为79%。在第二项研究中，人类评分者扮演学习者的角色，评估解释是否符合自身的信息需求。在所有教育背景中，用户认为GPT-4生成的解释对其信息需求的适配度平均比普通人类策划的解释低20%。此外，自动化评估指标显示，不同语言模型家族针对不同信息需求生成的解释在年级层级上难以区分，这限制了它们的教学效果。
英文摘要：
Language models today are widely used in education, yet their ability to tailor responses for learners with varied informational needs and knowledge backgrounds remains under-explored. To this end, we introduce ELI-Why, a benchmark of 13.4K "Why" questions to evaluate the pedagogical capabilities of language models. We then conduct two extensive human studies to assess the utility of language model-generated explanatory answers (explanations) on our benchmark, tailored to three distinct educational grades: elementary, high-school and graduate school. In our first study, human raters assume the role of an "educator" to assess model explanations' fit to different educational grades. We find that GPT-4-generated explanations match their intended educational background only 50% of the time, compared to 79% for lay human-curated explanations. In our second study, human raters assume the role of a learner to assess if an explanation fits their own informational needs. Across all educational backgrounds, users deemed GPT-4-generated explanations 20% less suited on average to their informational needs, when compared to explanations curated by lay people. Additionally, automated evaluation metrics reveal that explanations generated across different language model families for different informational needs remain indistinguishable in their grade-level, limiting their pedagogical effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
