PEARL: 使用教学对齐强化学习训练苏格拉底式导师
PEARL: Training Socratic Tutors with Pedagogically Aligned Reinforcement Learning
AI总结：
提出PEARL框架，通过可控学生模拟器、生成式奖励模型和稳定多目标强化学习，训练苏格拉底式教学代理，在多个基准上达到开源模型最佳性能并与专有模型竞争。
摘要：
大型语言模型（LLM）在教育辅导方面展现出潜力，但有效的辅导不仅仅是解决问题：它必须提供渐进的苏格拉底式引导，并在多轮交互中平衡多个教学目标。然而，由于学生模拟的保真度有限且可控性弱、教学奖励建模不明确以及多目标优化不稳定，训练这样的导师仍然具有挑战性。为克服这些限制，我们提出了PEARL，一个教学对齐的强化学习框架，用于训练苏格拉底式教学代理，包含三个关键组件。首先，我们引入了一个可控的学生模拟器，将潜在认知状态与响应生成解耦，以模拟多样的能力和误解。其次，我们开发了一个生成式奖励模型，联合评估教学质量和目标正确性以进行策略优化。最后，我们提出了一种稳定的多目标强化学习方案，在每个维度内离散化奖励并跨维度聚合归一化优势，防止高方差目标主导更新。在多个基准上的实验表明，尽管仅使用30B策略模型，PEARL在开源模型中取得了最佳性能，并与领先的专有LLM保持竞争力。
英文摘要：
Large Language Models (LLMs) show strong potential as educational tutors. Existing approaches typically train them to solve problems and provide correct answers, but this problem-solving-centered paradigm overlooks key requirements of effective tutoring: progressive guidance and the coordination of multiple pedagogical objectives across multi-turn interactions. Developing such tutors remains challenging because student behavior varies substantially with individual knowledge states, pedagogical effectiveness depends on multiple factors beyond final-answer correctness, and coordinating these objectives over tutor-student interactions is inherently difficult. To address these challenges, we propose PEARL, a PEdagogically Aligned Reinforcement Learning framework for training Socratic tutoring agents. First, we introduce a controllable student simulator that disentangles latent cognitive states from response generation, enabling simulation of diverse abilities and misconceptions. Second, we develop a pedagogically aligned reward model that jointly assesses pedagogical quality and objective correctness. Finally, we propose a stable multi-objective reinforcement learning approach that balances competing pedagogical objectives during tutor training. Experiments across multiple benchmarks show that PEARL performs competitively against tutoring-specific open-source systems and leading proprietary LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
