DreOPD：用于流匹配模型的退化参考外推式在线策略蒸馏
DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models
浏览论文内容
中文总结 AI 辅助
本研究提出用于流匹配模型的DreOPD方法，将隐式奖励外推转为闭式速度回归并引入退化参考强化对比，在单/多教师设置下，其平均性能优于OPD及多任务RL基线，多数指标超专用教师。
中文摘要 AI 辅助
流匹配模型目前是图像生成的主流方法，但它向多样化下游场景的适配通常依赖训练后调整，这可能导致特定任务优化目标间的冲突。强化学习可实现对原始模型之外特定任务奖励的直接优化，但轨迹级优化可能产生高方差梯度和跨任务干扰。在线策略蒸馏（OPD）能为学生模型的 rollout 提供密集且稳定的监督，不过传统的教师匹配仍基于模仿。我们提出 DreOPD，一种用于流匹配模型的退化参考外推式 OPD 方法，它衔接了上述两种范式。DreOPD 将隐式奖励外推转换为闭式速度回归，实现了兼具 OPD 稳定性的外推式训练后调整；还采用轻度退化的参考来强化教师-参考对比，从而得到更清晰的外推方向。在单教师和多教师设置上的实验表明，DreOPD 的平均性能优于 OPD 和多任务强化学习基线，且在多数指标上超过专用教师模型。
英文摘要
Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose DreOPD, a Degraded-reference extrapolative OPD method for flow-matching models that bridges these two paradigms. Our DreOPD converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that DreOPD outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.
发表机构
- Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
- Zhejiang University（浙江大学）
- Harbin Institute of Technology（哈尔滨工业大学）
机构由 AI 辅助整理，请以论文原文为准。
