DoubleFree:利用探索历史实现无rollout的多任务大语言模型训练
DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training
AI总结:
本研究提出DoubleFree方法,复用领域RL的历史轨迹实现无rollout的多任务大语言模型训练,仅需最小改动现有框架,在Qwen3系列等模型上大幅提升巩固速度、降低GPU耗时并保持多领域性能。
摘要:
多任务后训练可通过独立强化学习(RL)培养领域专家,并通过多教师在线策略蒸馏(MOPD)巩固其能力。但标准MOPD会生成在线策略学生响应,并在整个巩固过程中保持多个教师在线,带来大量系统成本。我们将该流程视为领域探索后接跨领域利用。领域RL不仅产生最终专家,还会产生训练过程中收集的轨迹。现有方法主要利用前者,而未充分利用后者。我们提出DoubleFree,该方法复用这些历史轨迹用于无rollout的巩固,并预计算教师监督以实现无在线教师的学生训练。为理解其有效性,我们分析了历史轨迹的更新何时可近似在线策略蒸馏更新,以及结合不同探索阶段的轨迹如何更好支持演化的学生。DoubleFree仅需对现有后训练框架做最小改动。在Qwen3-4B、Qwen3-8B和SmolLM3-3B的数学、代码和指令跟随任务上,DoubleFree实现了2.99-5.75倍的巩固速度提升,GPU小时数减少80.9%-90.1%,同时保持相当的多领域性能。
英文摘要:
Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$-$5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$-$90.1\%$, while maintaining comparable multi-domain performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。