arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.28396cs.LGcs.AI
收藏

ADWIN: 用于视野感知在线策略蒸馏的自适应窗口

ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation

School of Computer Science, Peking University(北京大学计算机科学系);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);LLM Department, Tencent(腾讯LLM部门)

Kun Liang, Chenming Tang, Clive Bai, Weijie Liu, Saiyong Yang, Yunfang Wu

AI总结:

提出ADWIN框架,通过自适应窗口动态调整在线策略蒸馏中的轨迹长度,在保持或提升准确率的同时,将训练成本降低最多4.1倍。

摘要:

在线策略蒸馏(OPD)通过沿着学生生成的轨迹训练学生模型,并利用教师反馈来迁移推理行为,但标准的全轨迹训练将每次更新与昂贵的完整轨迹绑定,并且可能过度分配监督到对当前学生边际价值较低的后半部分。我们通过有用监督视野重新审视这一假设:学生引起的轨迹可能偏离教师偏好的延续,而对齐的前缀可能已经保留了长视野OPD更新方向。我们提出ADWIN,一种用于OPD的自适应窗口框架,将轨迹长度视为在线可接受性决策,在短的教师锚定前缀上训练,同时使用延迟的全轨迹探测来审计前缀与全轨迹的对齐情况,并通过陈旧性控制自适应调整下一视野。在数学和代码推理基准测试中,包括单任务、多任务和强到弱设置,ADWIN在全轨迹OPD和基于前缀的基线方法上改善了准确率与计算成本的权衡,将端到端训练成本降低最多4.1倍,同时达到相当或更好的准确率。

英文摘要:

On-policy distillation (OPD) transfers reasoning behavior by training a student on teacher feedback along student-generated trajectories, but standard full-rollout training ties every update to a costly completion and can over-allocate supervision to late positions with low marginal value for the current student. We revisit this assumption through the useful supervision horizon: student-induced rollouts can drift from teacher-preferred continuations, while aligned prefixes may already preserve the long-horizon OPD update direction. We propose ADWIN, an adaptive-window framework for OPD that treats rollout length as an online admissibility decision, training on short teacher-anchored prefixes while using delayed full-rollout probes to audit prefix--full alignment and adapt the next horizon with staleness control. Across math and code reasoning benchmarks in single-task, multi-task, and strong-to-weak settings, ADWIN improves the accuracy--compute trade-off over full-rollout OPD and prefix-based baselines, reducing end-to-end training cost by up to 4.1 times while achieving comparable or better accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑