SPEAR：通过强化学习中的序列符号对齐提炼领域自适应推理骨架

中文摘要 AI 辅助

基于强化学习的知识蒸馏具备将复杂推理从教师模型迁移至学生模型的潜力，但目前面临关键两难困境：研究者需在稀疏的基于结果的奖励（提供的逻辑指导不足）与昂贵的神经过程奖励模型（PRMs，用于获取密集信号）之间做出选择。我们通过引入SPEAR（Symbolic Process Evaluation and Alignment Reward，符号过程评估与对齐奖励）解决该问题，这是一种无训练、即插即用的序列级在线策略蒸馏过程奖励方法。SPEAR将自然语言推理轨迹投影为领域自适应的符号里程碑，为过程级推理对齐提供高效代理。通过利用最长公共子序列（LCS）对齐学生探索与教师里程碑，SPEAR提供密集、感知顺序的奖励信号，在无需外部神经验证器的情况下强制逻辑一致性。我们在数学、科学和常识推理任务上的实验表明，SPEAR通过序列级蒸馏结合高效密集过程奖励，有效缩小了学生与教师模型之间的推理差距。我们的代码和数据可在该https URL获取。

英文摘要

Reinforcement learning-based knowledge distillation has the potential to transfer complex reasoning from teacher to student models, yet it currently faces a critical dilemma: researchers must choose between sparse outcome-based rewards, which provide insufficient logical guidance, or expensive neural Process Reward Models (PRMs) for dense signals. We resolve this by introducing SPEAR (Symbolic Process Evaluation and Alignment Reward), a training-free and plug-and-play process reward method for sequence-level on-policy distillation. SPEAR projects natural-language reasoning traces into domain-adaptive symbolic milestones, providing an efficient proxy for process-level reasoning alignment. By utilizing the longest common subsequence (LCS) to align student explorations with teacher milestones, SPEAR provides a dense, order-aware reward signal that enforces logical consistency without the need for an external neural verifier. Our experiments across math, science, and commonsense reasoning tasks demonstrate that SPEAR effectively bridges the reasoning gap between student and teacher models via sequence-level distillation with efficient dense process rewards. Our code and data are available at: https://github.com/zhuochunli/SPEAR.

发表机构

• University of Pittsburgh（匹兹堡大学）
• University of Connecticut（康涅狄格大学）

机构由 AI 辅助整理，请以论文原文为准。