大语言模型的教学对齐
Pedagogical Alignment of Large Language Models
AI总结：
针对大语言模型在教育场景中直接给答案不符合教学要求的问题，提出用合成数据构建数据集的人类偏好学习方法实现教学对齐，还提出基于困惑度的评估指标，实验证明该方法优于监督微调。
摘要：
大语言模型（LLM）在未经过教学微调的情况下应用于教育场景时，往往会直接给出答案，而非引导学生完成解题过程。这种方式不符合教学最佳实践，也限制了其作为教育工具的有效性。我们将训练LLM以模拟有效教学策略的目标称为“教学对齐（pedagogical alignment）”。本文研究了基于人类偏好学习（LHP）算法以实现该对齐目标。此过程中的一个核心挑战是缺乏可指导对齐的高质量偏好数据集。为解决这一问题，我们提出了一种利用合成数据生成技术构建大规模数据集的新方法，无需耗时且成本高昂的人工标注。基于该数据集，我们使用Llama和Mistral模型开展的实验表明，LHP方法优于标准监督微调（SFT），分别将教学对齐准确率提升了13.1%和8.7%。现有的评估方法也缺乏足够的量化指标来衡量LLM的教学对齐程度。为填补这一空白，我们提出了基于困惑度的新型指标，可量化LLM提供脚手架式引导而非直接答案的倾向，为教学对齐提供了可靠的衡量标准。我们的分析有力证明了在优化LLM行为方面LHP方法优于SFT，凸显了LHP方法在让LLM更好地契合教育目标、打造有效学习体验方面的潜力。代码和模型可通过此处获取：https://github.com/luffycodes/Tutorbot-Spock。
英文摘要：
Large Language Models (LLMs), when used in educational settings without pedagogical fine-tuning, often provide immediate answers rather than guiding students through the problem-solving process. This approach falls short of pedagogically best practices and limits their effectiveness as educational tools. We term the objective of training LLMs to emulate effective teaching strategies as `pedagogical alignment.' In this paper, we investigate Learning from Human Preferences (LHP) algorithms to achieve this alignment objective. A key challenge in this process is the scarcity of high-quality preference datasets to guide the alignment. To address this, we propose a novel approach for constructing a large-scale dataset using synthetic data generation techniques, eliminating the need for time-consuming and costly manual annotation. Leveraging this dataset, our experiments with Llama and Mistral models demonstrate that LHP methods outperform standard supervised fine-tuning (SFT), improving pedagogical alignment accuracy by 13.1% and 8.7% respectively. Existing evaluation methods also lack quantitative metrics to adequately measure the pedagogical alignment of LLMs. To address this gap, we propose novel perplexity-based metrics that quantify LLMs' tendency to provide scaffolded guidance versus direct answers, offering a robust measure of pedagogical alignment. Our analysis provides compelling evidence for the superiority of LHP methods over SFT in optimizing LLMs' behavior, underscoring the potential of LHP methods in better aligning LLMs with educational objectives and fostering effective learning experiences. Code and models are available \href{https://github.com/luffycodes/Tutorbot-Spock}{here}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
