AI 中文总结
针对在线策略蒸馏的特权幻觉问题，本文提出双锚定策略蒸馏框架，通过两级锚定机制缓解信息不对称，在Qwen3-4B等模型上显著提升性能。
AI 中文摘要
在线策略（自）蒸馏（OPSD）越来越多地用于语言模型的后训练，它利用特权信息增强教师模型，但会引发特权幻觉问题：学生模型学习到依赖特权信息的行为，而在推理阶段无法从自身上下文复现该行为，最终导致性能下降。本文指出，OPSD失败的根本原因在于特权教师与推理阶段学生之间的信息不对称。为解决该问题，本文提出双锚定策略蒸馏（DAPD）框架，包含两级锚定机制：双路径锚定（DPA）引入自条件桥，沿两条信息匹配路径对齐参考行为与rollout行为，防止特权依赖行为传递给推理阶段学生；双源锚定（DSA）在参考到rollout及rollout到参考两个方向应用上述路径，减少对特权参考引导的依赖同时保留正确性监督。大量实验表明，DAPD可显著缓解特权幻觉，在Qwen3-4B上平均比OPSD提升2.00个百分点，且增益随模型规模扩大而保持，4B模型提升2.69，32B模型提升2.78。
英文摘要
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.
