大型语言模型能否匹敌智能导学系统的适应性？一项基准评测研究
Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study
AI总结：
本文通过提示变体框架在75个真实导学场景中评测Llama3-8B、Llama3-70B和GPT-4o的教学适应性，发现当前LLM难以匹敌智能导学系统的适应性，并开源了可复现的评测方法。
摘要：
大型语言模型（LLMs）有望成为动态教学辅助工具。然而，目前尚不清楚LLMs能否复现智能导学系统（ITS）的适应性——在ITS中，学生知识和教学策略是被显式建模的。我们提出了一种提示变体框架，用于在来自一个ITS的75个真实导学场景中评估LLM生成的教学动作的适应性和教学合理性。我们系统地从提示中移除关键上下文成分（例如学生错误和知识成分），以生成每个场景的变体。三个代表性LLM（Llama3-8B、Llama3-70B和GPT-4o）生成了1,350个教学动作。我们使用文本嵌入和随机化检验来衡量每个上下文特征的省略如何影响LLM的输出（适应性），并使用一个经过验证的导学教师训练分类器来评估响应质量（教学合理性）。令人惊讶的是，即使是表现最好的模型也只能勉强模仿ITS的适应性。具体而言，Llama3-70B对学生错误表现出统计上显著的适应性。尽管Llama3-8B给出的建议在教学合理性得分上高于其他模型，但它在指令遵循行为（包括输出格式）方面存在困难。相比之下，GPT-4o能够可靠地遵循指令，但倾向于提供过于直接的反馈，这与有效导学相偏离——有效导学应通过开放式问题引导学习者以探查其知识。鉴于这些结果，我们讨论了当前基于LLM的导学不太可能产生可与已知有效的ITS导学相媲美的学习收益。通过我们的开源基准评测代码，我们贡献了一种可复现的方法，用于评估LLM的教学适应性和执行忠实度。
英文摘要：
Large Language Models (LLMs) hold promise as dynamic instructional aids. Yet, it remains unclear whether LLMs can replicate the adaptivity of intelligent tutoring systems (ITS)--where student knowledge and pedagogical strategies are explicitly modeled. We propose a prompt variation framework to assess LLM-generated instructional moves' adaptivity and pedagogical soundness across 75 real-world tutoring scenarios from an ITS. We systematically remove key context components (e.g., student errors and knowledge components) from prompts to create variations of each scenario. Three representative LLMs (Llama3-8B, Llama3-70B, and GPT-4o) generate 1,350 instructional moves. We use text embeddings and randomization tests to measure how the omission of each context feature impacts the LLMs' outputs (adaptivity) and a validated tutor-training classifier to evaluate response quality (pedagogical soundness). Surprisingly, even the best-performing model only marginally mimics the adaptivity of ITS. Specifically, Llama3-70B demonstrates statistically significant adaptivity to student errors. Although Llama3-8B's recommendations receive higher pedagogical soundness scores than the other models, it struggles with instruction-following behaviors, including output formatting. By contrast, GPT-4o reliably adheres to instructions but tends to provide overly direct feedback that diverges from effective tutoring, prompting learners with open-ended questions to gauge knowledge. Given these results, we discuss how current LLM-based tutoring is unlikely to produce learning benefits rivaling known-to-be-effective ITS tutoring. Through our open-source benchmarking code, we contribute a reproducible method for evaluating LLMs' instructional adaptivity and fidelity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
