arXiv 2601.145602026-01-21cs.CL

奖励模型如何思考：在教育中整合教学推理和思考奖励

Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education

Chosun University（昌原大学）；Korea University（韩国大学）；Seoul National University（首尔国立大学）；Korea Institute for Curriculum and Evaluation（韩国课程与评价研究院）；Upstage；Indiana University Bloomington（印第安纳大学布卢明顿分校）；Nanyang Technological University（南洋理工大学）

Unggi Lee, Jiyeong Bae, Jaehyeon Park, Haeun Park, Taejun Park, Younghoon Jeon, Sungmin Cho, Junbo Koh, Yeil Jeong, Gyeonggeon Lee

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出PedagogicalRL-Thinking框架，通过教学推理提示和思考奖励方法，提升LLM在教育场景中的教学推理能力和结构化决策能力。

摘要：

大型语言模型（LLMs）正越来越多地被部署为智能辅导系统，但针对教育情境优化LLMs的研究仍然有限。最近的研究提出了使用强化学习方法训练LLM辅导系统，但这些方法仅关注优化可见响应，而忽略了模型的内部思考过程。我们引入了PedagogicalRL-Thinking框架，通过两种新颖的方法将教学对齐扩展到教育中的推理LLMs：（1）教学推理提示，通过特定领域的教育理论引导内部推理，而不是通用指令；（2）思考奖励，明确评估并强化模型推理轨迹的教学质量。我们的实验表明，领域特定、理论支撑的提示优于通用提示，而思考奖励在与教学提示结合时效果最佳。此外，仅在数学辅导对话中训练的模型在训练中未见过的教育基准上表现更佳，同时保持了基础模型的事实知识。我们的定量和定性分析表明，教学思考奖励会产生系统性的推理轨迹变化，使辅导者的思考过程中增加教学推理并更加结构化的教学决策。

英文摘要：

Large language models (LLMs) are increasingly deployed as intelligent tutoring systems, yet research on optimizing LLMs specifically for educational contexts remains limited. Recent works have proposed reinforcement learning approaches for training LLM tutors, but these methods focus solely on optimizing visible responses while neglecting the model's internal thinking process. We introduce PedagogicalRL-Thinking, a framework that extends pedagogical alignment to reasoning LLMs in education through two novel approaches: (1) Pedagogical Reasoning Prompting, which guides internal reasoning using domain-specific educational theory rather than generic instructions; and (2) Thinking Reward, which explicitly evaluates and reinforces the pedagogical quality of the model's reasoning traces. Our experiments reveal that domain-specific, theory-grounded prompting outperforms generic prompting, and that Thinking Reward is most effective when combined with pedagogical prompting. Furthermore, models trained only on mathematics tutoring dialogues show improved performance on educational benchmarks not seen during training, while preserving the base model's factual knowledge. Our quantitative and qualitative analyses reveal that pedagogical thinking reward produces systematic reasoning trace changes, with increased pedagogical reasoning and more structured instructional decision-making in the tutor's thinking process.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。