AI 中文总结
该研究针对在线策略蒸馏中偏差规则无法判断错误前缀是否可纠正的问题，提出反事实可恢复性方法，通过分支分类状态，实现更强的监督效果与性能提升。
AI 中文摘要
在线策略蒸馏（OPD）对学生访问的轨迹进行监督，但基于偏差的规则无法确定错误前缀是否仍可纠正。我们将该决策建模为反事实可恢复性，通过预算匹配的教师延续和回退分支重放每个错误状态。根据它们的相对成功，状态被分为可恢复、不可逆转但可避免或模糊三类，这些标签指导训练是否保留、回退或常规监督相应轨迹。在AIME分支诊断中，可恢复状态的平均延续减回退效应为0.185，不可逆转但可避免状态的该值为-1.000，显示出相反的干预偏好。一种由分支推导的可恢复性代理实现了1.000的AUC，显著优于仅用偏差的0.392。在冻结评估中，可恢复性感知控制取得了记录的最强性能，在保留的AIME2025上达到0.578的成功率，而最佳基准为0.517；还将AIME2024-2025的average@32从0.2656提升至0.3125，GPQA-Diamond的average@32从0.2702提升至0.3070。组件消融进一步表明，保留教师可纠正前缀提供了最大的单独贡献。这些发现确立了可恢复性作为OPD中选择性监督的结果基础决策变量。
英文摘要
On-policy distillation (OPD) supervises student-visited trajectories, yet divergence-based rules cannot determine whether an erroneous prefix remains correctable. We formulate this decision as counterfactual recoverability and replay each error state through budget-matched teacher-continuation and rollback branches. Based on their relative success, states are categorized as recoverable, irreversible-but-avoidable, or ambiguous, and these labels guide whether training retains, rolls back, or conventionally supervises the corresponding trajectory. On AIME branch diagnostics, the mean continuation-minus-rollback effect is 0.185 for recoverable states and -1.000 for irreversible-but-avoidable states, demonstrating opposite intervention preferences. A branch-derived recoverability proxy achieves an AUC of 1.000, substantially outperforming divergence alone at 0.392. Across frozen evaluations, recoverability-aware control achieves the strongest recorded performance, reaching 0.578 success on held-out AIME2025 compared with 0.517 for the best baseline. It also improves AIME2024-2025 average@32 from 0.2656 to 0.3125 and GPQA-Diamond average@32 from 0.2702 to 0.3070. Component ablations further show that retaining teacher-correctable prefixes provides the largest individual contribution. These findings establish recoverability as an outcome-grounded decision variable for selective supervision in OPD.
Commentsfalse information
