AI 中文总结
针对标准在线自蒸馏（OPSD）未充分利用 rollout 时间结构的局限，提出 DASH 方法，通过自适应传播门调整 token 级监督权重，在三类数学推理基准、三种模型规模上均优于 vanilla OPSD，且无需额外前向传播。
AI 中文摘要
带可验证奖励的强化学习（RLVR）利用自动可验证的结果信号提升大语言模型的推理能力，但这些信号通常是稀疏的，且为序列级别的。在线自蒸馏（OPSD）通过在学生模型访问的前缀处查询特权教师模型，并提供密集的 token 级分布监督，缓解了这种稀疏性。尽管这种密集监督减轻了信号稀疏性，但我们发现标准 OPSD 仍未充分利用 rollout 的时间结构：它为每个局部 divergence 分配相同的系数，无论其位置或所属的 divergence 序列如何。在在线自回归生成中，相同的 divergence 幅度可能遵循不同的 discrepancy 历史，反映出教师与学生之间不匹配的不同演变。由于仅局部标量无法区分这些时间上下文，标准 OPSD 无法将其 token 级权重适配到已实现的 discrepancy 序列。为解决此局限，我们提出 divergence-adaptive 监督视界（DASH）：DASH 将每个局部蒸馏信号与序列级均值之间的差距映射为自适应传播门，随后利用这些门控制反向多步聚合。通过这种方式，DASH 根据生成过程中局部 divergence 的演变调整 token 级监督权重。在三个数学推理基准、三种模型规模上的实验表明，DASH 在所有基准、所有三种规模上均优于我们匹配的 vanilla OPSD 重运行版本；DASH 复用 OPSD 已计算的教师和学生分布，因此增益无需额外的教师或学生前向传播。代码：this https URL
英文摘要
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs. In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student. Since the local scalar alone cannot distinguish these temporal contexts, standard OPSD cannot adapt its token-level weights to the realized discrepancy sequence. To address this limitation, we propose Divergence-Adaptive Supervision Horizons (DASH). DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate and then uses these gates to control backward multi-step aggregation. By doing so, DASH adjusts token-level supervision weights according to how local divergences evolve during generation. Experiments on three mathematical reasoning benchmarks across three model scales show that DASH improves over our matched vanilla OPSD reruns on every benchmark at all three scales. DASH reuses the teacher and student distributions that OPSD already computes, so the gains require no additional teacher or student forward pass. Code: https://github.com/DBtxy/DASH-OPSD
Comments17 pages, 4 figures, 9 tables. Code at https://github.com/DBtxy/DASH-OPSD
