基石还是绊脚石?解读在线策略蒸馏中的岩石令牌
Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
AI总结:
本文研究在线策略蒸馏中持续高损失的“岩石令牌”,发现它们虽占据大量梯度但功能贡献微弱,提出绕过这些令牌可简化对齐过程。
摘要:
尽管近期关于可验证奖励强化学习(RLVR)的研究表明,一小部分关键令牌不成比例地驱动推理增益,但在线策略蒸馏(OPD)中类似的令牌级理解仍未探索。本文研究了高损失令牌——在OPD的逐令牌KL目标下,作为师生不匹配的最直接信号,根据现有研究,这些令牌应随着训练收敛而逐渐减少;然而,我们的实证分析显示并非如此。即使在OPD训练达到明显饱和后,仍有大量令牌持续表现出高损失;我们将这些令牌称为“岩石令牌”,它们可占生成输出中高达18%的令牌。我们的研究揭示了两个令人惊讶的悖论。首先,尽管这些令牌的高出现频率提供了不成比例的大份额总梯度范数,但岩石令牌本身在整个训练过程中保持停滞,抵抗教师驱动的修正。其次,通过因果干预,我们发现这些令牌对模型的实际推理性能贡献可忽略不计。这些发现表明,大量优化带宽被花费在学生模型无法或无需内化的结构和话语残差上。通过解构这些动态,我们证明策略性地绕过这些“绊脚石”可以显著简化对齐过程,挑战了统一令牌权重的必要性,并为大规模模型蒸馏提供了更高效的范式。
英文摘要:
While recent work in Reinforcement Learning with Verifiable Rewards (RLVR) has shown that a small subset of critical tokens disproportionately drives reasoning gains, an analogous token-level understanding of On-Policy Distillation (OPD) remains largely unexplored. In this work, we investigate high-loss tokens, a token type that--as the most direct signal of student-teacher mismatch under OPD's per-token KL objective--should progressively diminish as training converges according to existing studies; however, our empirical analysis shows otherwise. Even after OPD training reaches apparent saturation, a substantial subset of tokens continues to exhibit persistently high loss; these tokens, which we term Rock Tokens, can account for up to 18\% of the tokens in generated outputs. Our investigation reveals two startling paradoxes. First, despite their high occurrence frequency providing a disproportionately large share of total gradient norms, Rock Tokens themselves remain stagnant throughout training, resisting teacher-driven corrections. Second, through causal intervention, we find that these tokens provide negligible functional contribution to the model's actual reasoning performance. These findings suggest that a vast amount of optimization bandwidth is spent on structural and discourse residuals that the student model cannot or need not internalize. By deconstructing these dynamics, we demonstrate that strategically bypassing these ``stumbling blocks'' can significantly streamline the alignment process, challenging the necessity of uniform token weighting and offering a more efficient paradigm for large-scale model distillation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。