发表机构
South China Normal University（华南师范大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对脏历史多回合在线策略自蒸馏的脆弱性，提出SMOPD方法，通过选择性掩码低熵令牌稳定训练，在LiC数据集上提升了Qwen3模型的SHARDED视图准确率，验证了令牌级不确定性的可靠性。
AI 中文摘要
脏历史rollout使多回合在线策略自蒸馏（OPSD）变得脆弱：一旦学生模型输出错误的中间回复，后续回合将以该回复为条件，而均匀蒸馏会将损失浪费在几乎没有校正信号的令牌上。我们提出SMOPD（面向在线策略蒸馏的选择性掩码，Selective Masking for On-Policy Distillation），一种用于多回合OPSD的仅损失稳定方法。对于每个生成的中间回合回复，SMOPD按学生熵对令牌位置排序，并从裁剪的广义Jensen-Shannon蒸馏损失中移除熵最低的20%；最终答案和全量保留损失保持不变。该设计针对令牌级不确定性而非粗粒度轨迹结果，不增加参数，且无推理时间开销。我们将SMOPD与正确性缩放变体对比，该变体使用最终答案正确性乘以常见的分离可靠性代理。在使用Qwen3模型的LiC数据集上，SMOPD在单种子1.7B、4B和8B模型的比较中，将SHARDED视图准确率提升了1.0-2.5个百分点；小型4B多种子检查显示，相比基线平均SHARDED增益为+1.7个百分点（双尾p=0.022）。添加结果标量在无掩码的1.7B模型上有害（-4.0个百分点），与掩码结合时仍依赖规模（4B时+1.3个百分点，1.7B时中性，8B时-0.5个百分点）。这些存档的汇总结果表明，在评估的脏历史OPSD场景中，令牌级不确定性是比标量最终答案正确性更可靠的稳定信号，而因果机制测试和更广泛的基准验证留待未来工作。
英文摘要
Dirty-history rollouts make multi-turn on-policy self-distillation (OPSD) brittle: once a student emits an erroneous intermediate reply, later turns are conditioned on that reply, and uniform distillation can spend loss on tokens that carry little corrective signal. We introduce SMOPD (Selective Masking for On-Policy Distillation), a loss-only stabilization method for multi-turn OPSD. For each generated middle-turn reply, SMOPD ranks token positions by student entropy and removes the lowest-entropy 20% from the clipped generalized Jensen-Shannon distillation loss; final-answer and FULL-preservation losses are unchanged. This design targets token-level uncertainty rather than coarse trajectory outcomes, adds no parameters, and has zero inference-time overhead. We compare SMOPD with a correctness-scaling variant that multiplies a common detached reliability proxy using final-answer correctness. On LiC with Qwen3 models, SMOPD improves SHARDED-view accuracy by 1.0-2.5 percentage points in single-seed 1.7B, 4B, and 8B comparisons, and a small 4B multi-seed check shows a +1.7pp mean SHARDED gain over baseline (two-tailed p = 0.022). Adding the outcome scalar is harmful without masking at 1.7B (-4.0pp) and remains scale-dependent when combined with masking (+1.3pp at 4B, neutral at 1.7B, and -0.5pp at 8B). These archived aggregate results suggest that token-level uncertainty is a more reliable stabilization signal than scalar final-answer correctness in this evaluated dirty-history OPSD setting, while leaving causal mechanism tests and broader benchmark validation to future work.
